Japanese and Translations for User Interface

[dotike]

Hi All,

We're eager to focus now on making a working Japanese translation for OPNsense, we did it for pfSense before, but it did not yet work out, (notes below).

- What's the state of internationalization in the OPNsense user interface?

- Is there a plan to use gettext, or is there some other plan?

- I see that all file locations are happily quite different for the UI, as you folks are shearing it away from FreeBSD /src, good work!  But in the meantime, I can only find internationalized files for a few .js files.  (File list noted below, for the record).

- Is it possible to add a Japanese International forum?

Thanks!

Best,
.ike





--
Our history:

Last year, the pfSense user interface had been translated completely into Japanese.  Yet, with no build tools at the time the work was completed, it was never really able to be used, and the work stalled out.

The bulk of our effort can be seen in this one .po file:
https://raw.githubusercontent.com/pfsense/pfsense/07cf6e6e0454dc730f2916e54bbf2b9a569fc812/usr/local/share/locale/ja/LC_MESSAGES/pfSense.po

The overwhelming majority of the translations were completed by Chie Taguchi.

We were able to build on top of the excellent work the Brazilian Portuguese translation, (pt_BR.ISO8859-1)- they did the heavy lifting using gettext for the interface internationalization some years ago- making the translation files simple to find, and start working with.

--
Files list:

In pfSense, the .po gettext translation files were here, for Portuguese (Brazil) and English translations,
<pfsense-repo>./usr/local/share/locale/pt_BR.ISO8859-1/LC_MESSAGES/pfSense.po
<pfsense-repo>./usr/local/share/locale/en/LC_MESSAGES/pfSense.pot

/usr/local/share/locale is not at all a good location for the UI translation files, as it resides in the FreeBSD src base.

Some place more naturally packaged with the UI:
<core>./src/www/themes/locale/
Plus,

The only remnants of pt_BR I can find in the code are currently:

$ find . -name '*pt_BR*' -print
./src/www/themes/opnsense/assets/javascripts/bootstrap-select/js/i18n/defaults-pt_BR.js
./src/www/themes/opnsense/build/js/i18n/defaults-pt_BR.js
./src/www/themes/opnsense/build/js/i18n/defaults-pt_BR.min.js
./src/www/themes/sample/assets/javascripts/bootstrap-select/js/i18n/defaults-pt_BR.js
./src/www/themes/sample/build/js/i18n/defaults-pt_BR.js
./src/www/themes/sample/build/js/i18n/defaults-pt_BR.min.js
$

[dotike]

こんにちは、みんな、

このマシンベースの翻訳はご容赦ください。

私たちは前にpfSenseのためにそれをやった、OPNsenseの作業日本語訳を作ることになりました集中に熱心だが、それはまだ（下記注）、うまくいきませんでした。

- OPNsenseのユーザーインターフェイスの国際化の状態は何ですか？

- gettextを使用するための計画がありますか、または他のいくつかの計画はありますか？

- あなたの人々は、FreeBSD / SRC、良い仕事から離れて、それをせん断しているように私は、すべてのファイルの場所は、UIのために喜んでかなり異なっていることがわかります！しかし一方で、私は数の.jsファイル用の国際化ファイルを見つけることができます。 （ファイルリストは、レコードのため、以下に記載）。

- それは、日本国際フォーラムを追加することは可能ですか？

ありがとう！

ベスト、
.ike

[AdSchellevis]

Hi .ike,

At the moment we don't have any other mechanisms in place for translation as there where in pfSense, but for the new components we're planning for a similar approach using language files (and hook it into a Volt template tag / supply some helpers for the ajax parts).
For the time being we've chosen to remove the language files, because we're not sure how complete they are and we're unable to validate the contents our selves.

I guess that the language support still works for most parts when the translation files are placed inside the correct directory (gettext seems quite standard), but if that doesn't work I can look into it for you. It's very nice to hear that you want a Japanese version, so when I can help just let me know.

When you have patches for the project,  just put a pull request on github and we'll place it in our standard release.  (as long as we may include them under the 2 clause bsd licence).

It's no problem to have a Japanese forum, I will open one for you right away.

Cheers,

Ad

[dotike]

Ad,

At the moment we don't have any other mechanisms in place for translation as there where in pfSense, but for the new components we're planning for a similar approach using language files (and hook it into a Volt template tag / supply some helpers for the ajax parts).

OK- interesting.  I have no idea what Volt templates are, I'll do some reading and see if I can grok it enough to help appropriately, (and not waste your time).

For the time being we've chosen to remove the language files, because we're not sure how complete they are and we're unable to validate the contents our selves.

Honestly, that's a good idea- the UI is a fresh start and new direction, the translations should be too.

And for the long-term, juggling to make sure the quality of the translations is "up to snuff", and tracks releases, becomes a different problem to tackle- we have discussed this for Japanese with several ideas to help automate human validation of changes over time- but too early for that talk now...

I guess that the language support still works for most parts when the translation files are placed inside the correct directory (gettext seems quite standard), but if that doesn't work I can look into it for you. It's very nice to hear that you want a Japanese version, so when I can help just let me know.

Oh!! Is there a place we can start with the .po files now?  That will kickstart getting the translation patches up to you fast, even if it's not the permanent place for the file.

(In pfSense, the english .po file was the base canonical reference for all other translations.)

When you have patches for the project,  just put a pull request on github and we'll place it in our standard release.  (as long as we may include them under the 2 clause bsd licence).

Delighted, and enthusiastically supporting the 2 clause bsd license for all work.

It's no problem to have a Japanese forum, I will open one for you right away.

どうもありがとう!  Thank you!

All Japanese specific talk I'll move over there, but I may post general internationalization implementation questions to this "main" list...

Best,
.ike

[AdSchellevis]

Hi .ike,

The Volt template stuff is work in progress, if you want to have an idea of where this is heading you can look at the Proxy code which is under development.
- [/usr/local/]opnsense/mvc/app/models/OPNsense/Proxy/
- [/usr/local/]opnsense/mvc/app/controllers/OPNsense/Proxy/
- [/usr/local/]opnsense/mvc/app/views/OPNsense/Proxy/

I will post some notes on our wiki pages on how to enable the new code in the current environment, some develop information is already there.

For the location of the new locale files for opnsense I would like to suggest the following base location:

/usr/local/opnsense/locale/


But if this location turns out to be less practical, I will refactor the code to make it work later on. If you have some translation files for me, I will alter the path settings in the current codebase to make sure you can start easily.

Cheers,

Ad

[dotike]

The Volt template stuff is work in progress, if you want to have an idea of where this is heading you can look at the Proxy code which is under development.
- [/usr/local/]opnsense/mvc/app/models/OPNsense/Proxy/
- [/usr/local/]opnsense/mvc/app/controllers/OPNsense/Proxy/
- [/usr/local/]opnsense/mvc/app/views/OPNsense/Proxy/

I will post some notes on our wiki pages on how to enable the new code in the current environment, some develop information is already there.

Sounds good, thank you- I'll dig into it after I get through a first cut at fresh .pot files,

For the location of the new locale files for opnsense I would like to suggest the following base location:

/usr/local/opnsense/locale/

But if this location turns out to be less practical, I will refactor the code to make it work later on.

Hrm...  I *think* this sounds good, but I'm not up to speed on where 'core' layers in to 'src', (I haven't made time to build the whole thing yet).  For now, just to get working I put the files in:

[core]/src/share/locale/

https://github.com/dotike/opnsense.core.ja_JP.UTF8/tree/96aa0a8a29abd9deb6f68df7ebc4edc6ea1dbe4c/src/share/locale

I believe that it's wise to keep the gettext .pot files outside of the UNIX base hier(7), and instead, in the sort of 'core hier(7)'- that keeps it them neatly isolated away from system bits.  If /usr/local/opnsense/share is that spot, then it sounds great to me!

If you have some translation files for me, I will alter the path settings in the current codebase to make sure you can start easily.

Even though our target is Japanese, I'll have a completely fresh English .pot file done soon enough, generated fresh directly from every gettext call in the current OPNsense 'core' codebase.
Once I'm through the slog of getting it done, we'll all have something to work with- and other languages (Japanese) can go to town and start the translation work- without being blocked by aggressive changes in 'core'.

https://github.com/dotike/opnsense.core.ja_JP.UTF8/tree/96aa0a8a29abd9deb6f68df7ebc4edc6ea1dbe4c/src/share/locale/en/LC_MESSAGES

I hope to be done with it this week, and will toss yall' a pull request when it's sane- (it's just grunt work atm...)

Best,
.ike

[AdSchellevis]

Great!

My plan was indeed to isolate our language files from the base system, to prevent overlap with the standard layout.

For now it's absolutely fine to implement everything into [core]/src/share/locale/, as soon as you have something we can pull in I will move them around to the new location and make it work.

Cheers,

Ad

[franco]

I'd like to move the files to its proper target destination before merging them to avoid thousands of lines of movement in the history. I like both suggested directories. Since it's only for the GUI we can certainly get away with /usr/local/opnsense/locale.

Ike, do you have any generation tools/scripts that maybe need to go into tools.git or core.git to make it easier for others to work on translations as well?

[dotike]

Hi Franco, All,

5860 English strings pulled from the OPNsense source,

https://github.com/opnsense/core/pull/144

Now I'm walking through changes made to the source in the last month, to apply the changes to the .pot.

I'd like to move the files to its proper target destination before merging them to avoid thousands of lines of movement in the history. I like both suggested directories. Since it's only for the GUI we can certainly get away with /usr/local/opnsense/locale.

Yiiiikes. I just submitted a PR to core, finally with the completed file.

I'll leave it to your best judgement if you want the file moved, I can certainly submit another PR.

Ike, do you have any generation tools/scripts that maybe need to go into tools.git or core.git to make it easier for others to work on translations as well?

Yikes- I'm afraid I don't.  All the stuff in my branch is hack-ish one-off awk/sed stuff, and far from complete or safe.  (I kept chipping away at edge cases in pulling the strings cleanly, until I had gotten the human-editing cleanup work down to something reasonable).

With that, I've got parsing tools, but they were just scaffolding.

--
One set of tools I will need to hack out soon: something to run via cron (or git push hooks), which can be used to notify translators of:
- changes to the source which did not make it into the canonical English .pot
  (a harder problem to do perfectly)
- lines in a respective language file which now differ from the English canonical file

Best,
.ike

[franco]

Hi Isaac,

great work so far! I merged the PR as is: file location is fine with me (we can always move though), the commit history is good and there were only ever edits to a new file.

Ike, do you have any generation tools/scripts that maybe need to go into tools.git or core.git to make it easier for others to work on translations as well?

Yikes- I'm afraid I don't.  All the stuff in my branch is hack-ish one-off awk/sed stuff, and far from complete or safe.  (I kept chipping away at edge cases in pulling the strings cleanly, until I had gotten the human-editing cleanup work down to something reasonable).

With that, I've got parsing tools, but they were just scaffolding.

What I'd like to set up is a make target `translate' that runs a script inside core.git:scripts/ which can regenerate the fill pot file so that anyone can potentially run this and we have clean diffs of what changed in the pot file eventually (no manual shift/edit). The script(s) can be as complicated as need be, perl/python/ruby, anything really that helps to make the generation simple and sustainable even by non-developers. I can certainly help with that. I know a few things about parsers.

One set of tools I will need to hack out soon: something to run via cron (or git push hooks), which can be used to notify translators of:
- changes to the source which did not make it into the canonical English .pot
  (a harder problem to do perfectly)
- lines in a respective language file which now differ from the English canonical file

The above might take care of most of that?


Cheers,
Franco