Issue 11129

Title
Process task by batch
Priority
feature
Status
resolved
Nosy list
ced, pokoli, resteve, reviewbot, roundup-bot
Assigned to
ced
Keywords
review

Created on 2022-01-11.17:01:46 by ced, last changed 5 months ago by roundup-bot.

Messages

New changeset 5590076da369 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/tryton-env/rev/5590076da369
New changeset e436b206f484 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/trytond/rev/e436b206f484
New changeset 0e99ca435498 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/sale_supply_production/rev/0e99ca435498
New changeset 6a2afcc843a5 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/sale_supply_drop_shipment/rev/6a2afcc843a5
New changeset 6ee901ad8be2 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/sale_supply/rev/6ee901ad8be2
New changeset 59893ff64214 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/sale_complaint/rev/59893ff64214
New changeset ca16222876a9 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/sale_amendment/rev/ca16222876a9
New changeset c64ccc6f426b by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/sale/rev/c64ccc6f426b
New changeset 9b1bb584c474 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/purchase_requisition/rev/9b1bb584c474
New changeset 05d2f87aa19c by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/purchase_amendment/rev/05d2f87aa19c
New changeset 047b6ad84bab by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/purchase/rev/047b6ad84bab
New changeset cdf35272db70 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/product_kit/rev/cdf35272db70
New changeset 6f92dc3a76b5 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/account_payment_stripe/rev/6f92dc3a76b5
New changeset a753b03fbbab by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/account_payment_braintree/rev/a753b03fbbab
New changeset 157b5f9e2545 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/account_invoice_stock/rev/157b5f9e2545
New changeset 6ed44b4db8d3 by Cédric Krier in branch 'default':
Add batch option to push queue tasks
https://hg.tryton.org/modules/account_invoice/rev/6ed44b4db8d3
Author: [hidden] (ced) Tryton committer Tryton translator
Date: 2022-01-11.17:17:17

I activate by default the batch on modules where it makes sense.

Author: [hidden] (pokoli) Tryton committer Tryton translator
Date: 2022-01-11.17:10:01

Is there any plan to activate the feature by default on some modules?

Author: [hidden] (ced) Tryton committer Tryton translator
Date: 2022-01-11.17:01:46

We often put in the queue a method call with a list of records. But often there is no strict need that all the records are being processed in the same transaction nor at the same time.
We can have performance issue when the list of records is very large (and if the method take locks on table or many records) and it takes time to process all of them. So often the system will be more reactive if if we process small batch of records instead of all of them.
So I propose to add a new contextual parameters to __queue__ calls which allow the queue to create many tasks (batches) for one call. The parameter will allow to just activate the feature (with a default batch size) or define the size for the batch. Of course the division will only happen if there are workers otherwise there is no benefit.

History
Date User Action Args
2022-03-09 00:59:34roundup-botsetmessages: + msg74497
2022-03-09 00:59:28roundup-botsetmessages: + msg74496
2022-03-09 00:59:24roundup-botsetmessages: + msg74495
2022-03-09 00:59:20roundup-botsetmessages: + msg74494
2022-03-09 00:59:16roundup-botsetmessages: + msg74493
2022-03-09 00:59:11roundup-botsetmessages: + msg74492
2022-03-09 00:58:55roundup-botsetmessages: + msg74491
2022-03-09 00:58:47roundup-botsetmessages: + msg74490
2022-03-09 00:58:39roundup-botsetmessages: + msg74489
2022-03-09 00:58:33roundup-botsetmessages: + msg74488
2022-03-09 00:58:24roundup-botsetmessages: + msg74487
2022-03-09 00:58:14roundup-botsetmessages: + msg74486
2022-03-09 00:58:10roundup-botsetmessages: + msg74485
2022-03-09 00:58:06roundup-botsetmessages: + msg74484
2022-03-09 00:58:04roundup-botsetmessages: + msg74483
2022-03-09 00:58:01roundup-botsetmessages: + msg74482
nosy: + roundup-bot
status: testing -> resolved
2022-02-13 01:07:27reviewbotsetmessages: + msg74078
2022-01-11 19:20:16restevesetnosy: + resteve
2022-01-11 17:22:38reviewbotsetmessages: + msg73200
nosy: + reviewbot
2022-01-11 17:17:17cedsetkeyword: + review
messages: + msg73199
reviews: 381661002
status: in-progress -> testing
2022-01-11 17:10:01pokolisetmessages: + msg73198
nosy: + pokoli
2022-01-11 17:01:46cedcreate