Tryton - Issues

 

Issue5907

Title unoconv conversion fails from within trytond
Priority bug Status deferred
Superseder Add pluggable report converter
View: 5032
Nosy List Timitos, alikefia, ced, christoph.larsen@synalinq.com, fabyc, nblock, oscar, perilla, pokoli, reviewbot, yangoon
Type Components trytond
Assigned To ced Keywords review
Reviews 34341002
View: 34341002

Created on 2016-09-25.07:41:22 by christoph.larsen@synalinq.com, last changed by pokoli.

Messages
msg30278 (view) Author: [hidden] (fabyc) (Tryton translator) Date: 2016-11-11.16:14:35
>Someone has tried to use unoconv as server?

>The issue with the current implementation is that any call to unoconv starts a new server to make the conversion. This creates conflicts on the instance of oo.

>We fixed that by:
>- launching unoconv as a standalone server (--listener)
>- change trytond calls to be client only (--no-launch)
>Since that change, unoconv is much more stable

>NB. The issue with that review is that no more server is possible and we are not sure to get conflicts gone since they are due to libreoffice.

I tested similar solution, but I still got errors sometimes like it is explained here: https://groups.google.com/d/msg/tryton/prztgxYjm0I/dqCLOPulCAAJ

Similar error like explained in msg28993.
msg29787 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.18:55:28
Indeed I will not add a loop because for now the patch will raise an issue in such case and also if there are not enough resources relaunching the process will not help.
msg29786 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.18:49:21
For what I understand, the failure are due to missing resources (maybe memory as we start a many processes).
So I will update the patch to make few retries in such case but I will also delay it until we have the mechanism to extend globally the Report class.
msg29783 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.18:05:03
On 2016-10-26 17:35, Korbinian Preisler wrote:
> I think we should allow to override the way how the conversion is to be done. with the planned override mechanism for base classes this should be doable easily.

Yes of course this will be a good improvement and by the way the
refactoring of Report from 3.6 defines a single method for format
conversion.

> I think it would be good to move the conversion out to some workers. Especially in bigger installations or installations with bigger documents this could be a good solution.

This will be done like for the redis cache. But by default, we need a
simple solution that works out of the box.
msg29779 (view) Author: [hidden] (perilla) Date: 2016-10-26.17:55:13
See this tool of Coopengo, maybe useful for test concurrency reports
https://github.com/coopengo/trytond-report-bench
msg29777 (view) Author: [hidden] (Timitos) Date: 2016-10-26.17:35:44
I think we should allow to override the way how the conversion is to be done. with the planned override mechanism for base classes this should be doable easily. I think it would be good to move the conversion out to some workers. Especially in bigger installations or installations with bigger documents this could be a good solution. For the future i think it would be good to have another report engine that focuses on creating pdfs for all reports that do not need to be convertable.
msg29776 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.16:55:03
On 2016-10-26 16:26, Ali Kefia wrote:
> does oo script work on your machine?

No because it is bashism.

> PS: even if master script leaves, execution continues

But you still count the wrong numbers.
msg29774 (view) Author: [hidden] (alikefia) Date: 2016-10-26.16:26:01
does oo script work on your machine?
PS: even if master script leaves, execution continues
msg29772 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.16:14:34
Maybe but I still do not buy your test script. If you look at the start-up script of soffice, it does many exec and one is oosplash.
Indeed you should wait for each jobs.
msg29767 (view) Author: [hidden] (alikefia) Date: 2016-10-26.15:30:44
Have pushed a kind of bench, if someone is interested to test:
https://github.com/coopengo/trytond-report-bench
msg29766 (view) Author: [hidden] (alikefia) Date: 2016-10-26.14:16:19
It should continue writing since jobs are launched as an async subshell
msg29765 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.14:01:50
I guess your script is not correctly waiting all the jobs.
soffice is not the executable.
msg29763 (view) Author: [hidden] (alikefia) Date: 2016-10-26.13:30:10
The problem is that version 4.x is the default on Debian (a usual distribution for deployment)
https://packages.debian.org/jessie/libreoffice

Tried on local and on a newer Office version via Docker, the result is the same (I have an image on my test.odt document)

Scripts are here: https://gist.github.com/alikefia/32b659deb96bc5d1eb4726ce0727177e

## Local:

~/c/s/pdf ❯❯❯ ./convert 10
converting 10 files
LibreOffice 4.2.8.2 420m0(Build:2)

8 => 1
4 => 1
6 => 0
2 => 0
3 => 0
1 => 0
9 => 0
7 => 0
5 => 0
10 => 0
8 pdf generated

## Newer version:

~/c/s/pdf ❯❯❯ docker run -it pdf 10
converting 10 files
LibreOffice 5.1.4.2 9b41a63f5af9c5b93dae98ccbfcda2d4ec09ef7a

6 => 1
9 => 77
5 => 0
4 => 0
10 => 0
3 => 0
2 => 0
1 => 0
8 => 0
7 => 0
8 pdf generated
msg29749 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.11:41:56
Version 4.2.8.2 is 2 years old, it is considered by LibreOffice as obsolete.
You should try with a more up to date version.
msg29748 (view) Author: [hidden] (alikefia) Date: 2016-10-26.11:36:20
Nothing on strerr nor on stdout
LibreOffice 4.2.8.2 420m0(Build:2)
msg29747 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.11:31:25
I think you should look at the error output to understand what is happening on your side.
msg29746 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.11:29:24
I tested with soffice 5.1.4.2 and I got all the pdf generated.
Of course it is a little bit slow on my small machine but it works.
msg29745 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-26.11:20:17
What do you mean by "lost calls"?
msg29744 (view) Author: [hidden] (alikefia) Date: 2016-10-26.11:14:53
I made some tests and there are lost calls

(test the difference with and without &)

#!/bin/bash

for i in $(seq 1 10)
do
  ( echo "$i" && soffice --headless --nolockcheck --nodefault --norestore --convert-to pdf --outdir ./out "./odt/test-$i.odt" &> /dev/null && echo "$i => $?" ) &
done
msg29697 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-25.18:29:14
I have tested three loops running at the same time converting 100 odt each one.
I have seen no conflicts so far and I see three soffice process running at the same time.
msg29696 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-25.18:15:04
On 2016-10-25 17:44, Ali Kefia wrote:
> - launching unoconv as a standalone server (--listener)
> - change trytond calls to be client only (--no-launch)
> Since that change, unoconv is much more stable

Still possible by provide a custom soffice script in the PATH.

> NB. The issue with that review is that no more server is possible and we are not sure to get conflicts gone since they are due to libreoffice.

This should be tested.
msg29695 (view) Author: [hidden] (alikefia) Date: 2016-10-25.17:44:54
Someone has tried to use unoconv as server?

The issue with the current implementation is that any call to unoconv starts a new server to make the conversion. This creates conflicts on the instance of oo.

We fixed that by:
- launching unoconv as a standalone server (--listener)
- change trytond calls to be client only (--no-launch)
Since that change, unoconv is much more stable

NB. The issue with that review is that no more server is possible and we are not sure to get conflicts gone since they are due to libreoffice.
New review34341002 at https://codereview.tryton.org/34341002/#ps20001
msg29376 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-14.13:37:19
Here is review34341002 that implements it.
It will be great to have feedback from different setup with different kind of loads (concurrent users etc.)
msg29372 (view) Author: [hidden] (ced) (Tryton committer) (Tryton translator) Date: 2016-10-14.12:28:09
Why not removing unoconv and instead just use the command line option of LibeOffice:

soffice --headless --convert-to pdf file.odt

Of course we will have to start a new process for each conversion but it is already the case with the default option.
But the main advantage is that it does not rely on the uno/pyuno API of LibreOffice which shows to be not very reliable.
msg29179 (view) Author: [hidden] (christoph.larsen@synalinq.com) Date: 2016-10-04.18:53:50
On 04/10/16 18:19, Oscar Andres Alvarez wrote:
> Oscar Andres Alvarez <oscar.alvarez.montero@gmail.com> added the comment:
>
> I can confirm this bug on tryton 4.0, when PDF format is setted
>
> ----------
> nosy: +oscar
>
> _______________________________________________
> Tryton issue tracker <issue_tracker@tryton.org>
> <https://bugs.tryton.org/issue5907>
> _______________________________________________
Thanks, Oscar. A major inconvenience, a I do not want to install 
LibreOffice locally on the workstation.
Imagine Africa, brittle infrastructure, poor access to reliable mains 
power, patchy internet cover and enthusiasitc, but poorly trained users: 
Less is more. PDF wins.

Chris
msg29171 (view) Author: [hidden] (oscar) (Tryton translator) Date: 2016-10-04.17:19:34
I can confirm this bug on tryton 4.0, when PDF format is setted
msg29095 (view) Author: [hidden] (pokoli) (Tryton committer) Date: 2016-09-29.18:59:27
> Mathias Behrle <mbehrle@m9s.biz> added the comment:
> 
>> Sergi Almacellas Abellana <sergi@koolpi.com> added the comment:
>>
>> @yangoon why you don't propose the patch to be included in trytond?
> 
> IIRC we did that yet for 2.2 and at that time it was not accepted.

I haven't found any issue on the bugtracker, could you please give some link so I can follup up?  Thanks
msg29094 (view) Author: [hidden] (yangoon) (Tryton translator) Date: 2016-09-29.18:28:05
> Sergi Almacellas Abellana <sergi@koolpi.com> added the comment:
> 
> @yangoon why you don't propose the patch to be included in trytond?

IIRC we did that yet for 2.2 and at that time it was not accepted.
msg29093 (view) Author: [hidden] (pokoli) (Tryton committer) Date: 2016-09-29.16:40:08
@yangoon why you don't propose the patch to be included in trytond?
msg29050 (view) Author: [hidden] (yangoon) (Tryton translator) Date: 2016-09-27.23:48:01
We found unoconv to not be thread safe. We are running with a listener and the patch at
https://gitlab.com/m9s/trytond-patches/blob/mbs-3.4/unoconv_retry.patch
without problems (usually we set retry to 5 in trytond.conf). Perhaps it can you help, too.
msg28993 (view) Author: [hidden] (christoph.larsen@synalinq.com) Date: 2016-09-25.07:41:22
Dear All,
I am running trytond 3.4 (latest minor version) on FreeBSD with the default unoconv pipe settings "unoconv = pipe,name=trytond;urp;StarOffice.ComponentContext" inside my configuration file.
LibreOffice headless version is 5.0.6.3.
I run trytond as user trytond.
If I run: su trytond -c "unoconv -c 'pipe,name=trytond;urp;StarOffice.ComponentContext' /tmp/test_document.odt", everything  works fine.
Using the default trytond report configuration with 'OpenDocument Text" as Template Extension and with empty Extension, I can print labels, which are produced via my local (workstation) LibreOffice instance.
Changing Extension to Portable Document Format" produces this error:
--------------------------------%<-----------------------------
Traceback (most recent call last):
  File "/site-packages/trytond/protocols/jsonrpc.py", line 150, in _marshaled_dispatch
    response['result'] = dispatch_method(method, params)
  File "/site-packages/trytond/protocols/jsonrpc.py", line 179, in _dispatch
    res = dispatch(*args)
  File "/site-packages/trytond/protocols/dispatcher.py", line 161, in dispatch
    result = rpc.result(meth(*c_args, **c_kwargs))
  File "/site-packages/trytond/report/report.py", line 144, in execute
    type, data = cls.parse(action_report, records, data, {})
  File "/site-packages/trytond/report/report.py", line 300, in parse
    data = cls.unoconv(data, report.template_extension, output_format)
  File "/site-packages/trytond/report/report.py", line 320, in unoconv
    raise Exception(stderrdata)
Exception: None
--------------------------------%<-----------------------------
This, although the direct deployment of unoconv works, see above. If at all possible, I would like to do away with the need to call LibreOffice locally on the workstation.
Am I missing something, or is this a bug? thanks a lot!
Chris
History
Date User Action Args
2017-02-10 13:30:54pokolisetsuperseder: + Add pluggable report converter, - Allow to Apply inheritance on models loaded in the pool
2016-11-17 12:42:43nblocksetnosy: + nblock
2016-11-11 16:14:35fabycsetmessages: + msg30278
2016-11-11 15:06:35fabycsetnosy: + fabyc
2016-11-10 11:31:36cedlinkissue6025 superseder
2016-10-26 18:55:28cedsetmessages: + msg29787
2016-10-26 18:49:22cedsetstatus: testing -> deferred
superseder: + Allow to Apply inheritance on models loaded in the pool
messages: + msg29786
2016-10-26 18:05:03cedsetmessages: + msg29783
2016-10-26 17:55:13perillasetnosy: + perilla
messages: + msg29779
2016-10-26 17:35:45Timitossetnosy: + Timitos
messages: + msg29777

Showing 10 items. Show all history (warning: this could be VERY long)