I18n plans. “Language Translation" for the plebs

@George any in the short/medium term?

Just asking because of this.

I have a huge epic planned that starts today and as always I don’t want to start if there is anything planned.

My approach will be based on json files and a humungous amount of custom formatters.

We don’t have specific localization support planned, but you can easy do it yourself with loading indeed json data as global app connect data and use your translations as data bindings.

You could use the JSON data source, but it is loaded asynchronous. So it might take time to be available.

A better approach is to have direct app connect global data variables in a js file that is directly included asfter dmxAppConnect.js

The App Connect global data can have any data that will be loaded directly on the app connect global level.

You can use it to share settings or store your localization there.

See the example of Patrick:

https://community.wappler.io/t/using-summernote-wysiwyg-editor/19855/24

Thanks for confirmation.

Interesting approach. I am thinking how I could integrate this with being able to crowdsource the translation of strings as platforms for that need to understand the format. Using a json file made that part easy.

Any thoughts/ideas?

Hey Jon,

I’ve done it via the database at the moment. I have a massive .sql file with all the language text in, and will pass it on to others to edit. The format is easier for a non techie to edit than JSON I think.

I may create a front end for language translators, which is much more easily done if the text is in a database than a JSON file.

The other advantage is if I have 1000 text segments and someone only translates 900 into Spanish, I have a little stored procedure that will display the other 100 in English, so no part of my app is without text… or if I add 3 new text segments then they will be there in English until my Spanish translator gets to do their work on those 3.

Oh, and you may want to change the title of this post to something that includes the words “Language Translation” as maybe people don’t know what I18n is!

Best wishes,
Antony.

SQL FILE EXAMPLES:

-- Contacts
INSERT INTO apptext (lang_code, index_, app_text) VALUES ("en-GB", 100, "Contacts");
INSERT INTO apptext (lang_code, index_, app_text) VALUES ("es-ES", 100, "contactos");
INSERT INTO apptext (lang_code, index_, app_text) VALUES ("de-DE", 100, "Kontakte");
INSERT INTO apptext (lang_code, index_, app_text) VALUES ("en-GB", 101, "Work with all your business contacts");

...
-- email_cc
INSERT INTO apptext (lang_code, index_, field_name, app_text) VALUES ("en-GB", 8550, "email_cc", "Email address which emails can be copied to"); -- Field Title
INSERT INTO apptext (lang_code, index_, field_name, app_text) VALUES ("en-GB", 8551, "email_cc", "Email address which emails can be copied to"); -- Field Description
-- INSERT INTO apptext (lang_code, index_, field_name, app_text) VALUES ("en-GB", 8552, "email_cc", "Email address..."); -- Input Placeholder
INSERT INTO apptext (lang_code, index_, field_name, app_text) VALUES ("en-GB", 8559, "email_cc", "If an email address is specified here, then emails sent from the system will be copied to this address. 
This can be useful if you wish to be sure messages have been sent.<br>
If you use this option, you may want to specify an address that is different from your usual address, such as emailcc@mydomain.com. 
You can the set up your email system such as gmail to automatically bypass your inbox and route these emails to a particular folder. 
For example, this is done in gmail via the Settings -> Folders option. It may also require you to ensure that an address such as emailcc at your domain will be forwarded to your email system."); -- Help Text

USE IN THE APP:

<dmx-value id="apptext" dmx-bind:value="get_apptext.data.get_apptext.toKeyedObject(`index_`, `app_text`)"></dmx-value>
...
<a ...>{{apptext.value.100}}</a>

Sounds familiar :smiley:

Regarding the use of db or a json file it’s just a matter of how you store content.

I actually was thinking more in a complete i18n framework that handles plurals, gender, interpolation, etc

These frameworks try solve some of the problems localization has.

Do you have all the strings translated for the three genders? Male, Female and undisclosed? What about the strings “No notifications”, “1 notification” and “2 notifications”. For all supported languages.

I want to get that part right before I embark myself with actually translating the content.

Maybe you should check services like:

https://www.i18next.com/

See how well their integration and API works, do some research and if all goes well we might integrate the best as App Connect Components / API’s in Wappler

2 Likes

I already did some research in the past and I haven’t found “the library”. Sure there a good options out there. But none that shouts “this is the right choice”. This is a complicated one and I do understand why you have been keeping away from it.

I started with my own mix and match by extracting concepts and ideas from i18next, projectfluent and of course app connect and custom formatters.

var translation = {
    "en": {
        "addedphotostostream_female": "added {{count}} photo to her stream",
        "addedphotostostream_female_plural": "added {{count}} photos to her stream",
        "addedphotostostream_male": "added {{count}} photo to his stream",
        "addedphotostostream_male_plural": "added {{count}} photos to his stream"
    },
    "es": {
        "addedphotostostream_female": "añadió {{count}} foto a su muro",
        "addedphotostostream_female_plural": "añadió {{count}} fotos a su muro",
        "addedphotostostream_male": "añadió {{count}} foto a su muro",
        "addedphotostostream_male_plural": "añadió {{count}} fotos a su muro"
    }
}

dmx.Formatter('string', 't', function (key, language, options) {
    if (options.context)
        key += "_" + options.context; console.log(key)
    if (options.count == 1)
        return (translation[language.substring(0, 2)][key]).replace("{{count}}", options.count)
    else
        return (translation[language.substring(0, 2)][key + "_plural"]).replace("{{count}}", options.count)
});
<h6 class="h3">Monica {{"addedphotostostream".t(browser.language, {context:"female", count: 1})}}</h6>
<h6 class="h3">Monica {{"addedphotostostream".t(browser.language, {context:"female", count: 3})}}</h6>
<h6 class="h3">David {{"addedphotostostream".t(browser.language, {context:"male", count: 1})}}</h6>
<h6 class="h3">David {{"addedphotostostream".t(browser.language, {context:"male", count: 3})}}</h6>

Will render in english:

image

And in spanish:

image

Of course this is just bare-bones but you get the idea. I need to define a better way of dealing with translation keys, locales and finish the logic.

i18next might be overkill when you just want to use 4 or 5 western languages.

Of course, if you plan to localize your app in 50 different languages then yes i18next seems like a good fit.

Anyway I will continue to iterate through this idea.

1 Like

Very nice :slight_smile: :+1:

After 24 hours reinventing the wheel I’m reverting back to the idea of using i18next

It just works out of the box by initializing it and using a custom formatter to handle the parameters that will be passed to i18next :smiley:

That way you can continue leveraging Wappler’s framework such as data bindings and formatters.

<script src="https://unpkg.com/i18next/dist/umd/i18next.min.js"></script>
<script>
i18next.init({
            lng: 'en',
            resources: {
                en: {
                    translation: {
                        "key": "{{what}} has {{count}} oranges",
                        "friend": "A friend",
                        "friend_male": "A boyfriend",
                        "friend_female": "A girlfriend",
                        "friend_male_plural": "{{count}} boyfriends",
                        "friend_female_plural": "{{count}} girlfriends"                    
                    }
                },
                es: {
                    translation: {
                        "key": "{{what}} tiene {{count}} naranjas",
                        "friend_male": "Un amigo",
                        "friend_female": "Una amiga",
                        "friend_male_plural": "{{count}} amigos",
                        "friend_female_plural": "{{count}} amigas"                    
                    }            
                }
            }
	}).then(dmx.Formatter('string', 't', function (key, options) {
		return i18next.t(key, options);
}));
</script>

The main difference is how you call the translation function.

While i18next expects you to pass the key and the options as function parameter we will use the standard AC formatter way and call the string object function and pass the options.

i18next expects:

i18next.t('key', options)

We call a custom formatter

"key".t(options)

The rest just works out of the box. You can configure your needs as per their configuration options

<dmx-value id="name" dmx-bind:value="'Lisa Smith'"></dmx-value>
<dmx-value id="count" dmx-bind:value="20"></dmx-value>


<p>{{ "key".t({ what: name.value, count: count.value }) }}</p>
<p>{{ "friend".t() }}</p>
<p>{{ ("friend".t({context: 'male', count: 1})).uppercase() }}</p>
<p>{{ "friend".t({context: 'female', count: 1}) }}</p>
<p>{{ "friend".t({context: 'male', count: 100}) }}</p>
<p>{{ "friend".t({context: 'female', count: 100}) }}</p>


<p>{{ "key".t({ what: name.value, count: count.value, lng: 'es' }) }}</p>
<p>{{ "friend".t({context: 'male', count: 1, lng: 'es'}) }}</p>
<p>{{ "friend".t({context: 'female', count: 1, lng: 'es'}) }}</p>
<p>{{ "friend".t({context: 'male', count: 100, lng: 'es'}) }}</p>
<p>{{ "friend".t({context: 'female', count: 100, lng: 'es'}) }}</p>

image

5 Likes

@George @Teodor I am progressing with the i18next integration but still having problems when I load translations through the i18next XHR api.

I’ve simplified as much as possible the code. It’s quite straightforward.

  1. I init the i18next in the head and make it use the xhr backend api which loads i18ntest.json
  2. Once the i18next component is initialized I set an AC formatter which will return the translated
    text.
  3. I then set a SC json datasource that will retrieve the exact same file.
  4. Last but not least I call to bindings. The first one looks for the key via the datasource and the second one invokes the custom formatter.

<html lang="en">

<head>
	<base href="/">
	<script src="dmxAppConnect/dmxAppConnect.js"></script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/i18next/19.4.4/i18next.min.js"></script>
	<script src="https://cdn.jsdelivr.net/gh/i18next/i18next-http-backend@1.0.8/i18nextHttpBackend.min.js"></script>
	<script>
		i18next
		.use(window.i18nextHttpBackend)
		.init({
			backend: {	
				loadPath: 'i18ntest.json'
			}
	}).then(dmx.Formatter('string', 't', function (key, options) {
		return i18next.t(key, options);
	}))
	</script>
</head>

<body is="dmx-app" id="i18ntest">
	<p><strong>SC JSON Data Source:</strong> {{trans.data['key']}}</p>
	<p><strong>i18next loadPath:</strong> {{'key'.t()}}</p>
	<dmx-json-datasource id="trans" is="dmx-serverconnect" url="i18ntest.json"></dmx-json-datasource>
</body>

</html>

This is the result.

Each flick is a reload browser action.

You can clearly see that the AC binding has no issues rendering the value from the JSON retrieved via Server Connect. But many times the JSON file retrieved via i18next library is not loaded in time for AC to pick up the data.

Both files are retrieved via HTTP request but as expected if you keep things in-house the data will be picked correctly. My suspicion is that the AC bindings will wait for the SC as it’s being referenced in the own binding(please correct me if I’m wrong). But as AC doesn’t know about the existence of the i18next library it will not wait for the file to download.

Any ideas how to overcome this? It needs to be via file as all Continuous Translation Management Systems work with them

Maybe @patrick should better check if we can integrate the i18next services better in an app connect component

That would be even better. As per my findings I don’t think it will take a lot of effort.

The only issue I have found so far is not knowing better the insides of App Connect :slight_smile:

That is why we have Patrick - the maker :slight_smile:

3 Likes

Hmmm… Calling those less familiar with terminology than you are “plebs” wasn’t quite what I had in mind! :slight_smile:

What can I say? I am a millennial and a gamer. It was bound to affect me at some point on top of my high horse! :joy:

We are all plebs to someone else.

All your plebs are belong to us.

1 Like

Sorry to bother you @patrick any tip so I can delay the evaluation of bindings after a file has been retrieved from the server(but not via SC).

The init function will resolve the promise once all translations are loaded.

As per their docs

So you should wait for init to complete (wait for the callback or promise resolution) before using the t function!

That’s why I added the custom formatter that calls the t function in the promise resolve.

But that is not enough(obviously) because the binding evaluation doesn’t know about this.

I suspect that one possible solution would be to create a backend plugin for i18next that retrieves the file via SC but not really sure if that would make the binding wait for the file to be loaded.

https://www.i18next.com/misc/creating-own-plugins#backend

Don’t put the formatter code in the then method, in the then method only call dmx.requestUpdate. The requestUpdate will evaluate all expressions and update the DOM.

dmx.Formatter('string', 't', function (key, options) {
  return i18next.t(key, options);
});

i18next
.use(window.i18nextHttpBackend)
.init({
  backend: {
    loadPath: 'i18ntest.json'
  }
})
.then(dmx.requestUpdate);
1 Like

Brilliant! It works. Thanks!
Adding that function to the secret toolbox :slight_smile:

Now I just need to avoid the i18next flickering between showing the key and evaluating the t function. But that’s a matter of hiding the body until it reevaluates all expressions unless you can come up with a more elegant solution.

So I set a global variable on resolve and queried it to show the body.

<html lang="en">

<head>
	<base href="/">

	<script src="dmxAppConnect/dmxAppConnect.js"></script>
	<script src="https://cdnjs.cloudflare.com/ajax/libs/i18next/19.4.4/i18next.min.js"></script>
	<script src="https://cdn.jsdelivr.net/gh/i18next/i18next-http-backend@1.0.8/i18nextHttpBackend.min.js"></script>
	<script>
		i18next
		.use(window.i18nextHttpBackend)
		.init({
			backend: {	
				loadPath: 'i18ntest.json'
			}
	}).then( function() {	
		dmx.global.set('translationLoaded', true)
		dmx.requestUpdate()		
	}		
		)
	dmx.Formatter('string', 't', function (key, options) {		
		return i18next.t(key, options);
	})
	</script>

</head>

<body is="dmx-app" id="i18ntest" dmx-show="translationLoaded">
	<p><strong>i18next loadPath:</strong> {{'key'.t()}}</p>
</body>

</html>

Which will prevent showing the first render and will avoid the flickering.

Is there a cleaner approach? Maybe there is already dmx variable changed by dmx.requestUpdate() I can query?

1 Like