Magento Integration (filters and batches)

Hi,

Not sure where to put this but I’ll try it here. I’ve been working with OroCRM (rc-1) and the OroCRM bridge (1.0.1) for a few days now and I’ve found some, bugs or an interesting implementation for synchronizing with Magento. This applies to filters being send and the batches from Oro(CRM).

First of all, I’ve created a channel for orders(in orocrm) and installed the OroCRM bridge (v1.0.1) on Magento’s side.
The version of OroCRM is RC-1.0.0 and the Magento version is 1.7.0.2 so fairly recent on both sides.

What we wanted to do was to synchronize the Orders from Magento to OroCRM for a specific store. The total orders are approximately 104,330 (for all stores). The store which we would like to sync has 41,911 orders. When start the sync OroCRM will iterate over all (104,330) orders instead of the expected 41,911 orders.

Looking through the code it will send the storeview as a filter to Magento (and is received correct by Magento) but still creates batches for all of the Orders, my question is why is it iterating over all of the orders instead of the 41,911?

The filters which are being applied in Oro as well as in Magento are the storeview filter and the created_at[from] filter. They seem to do the job very well, except that I’ve expected a different working of the created_at filter. Since Oro is sending the filters I’m finding it strange that the created_at filter is always the “sync date” from the Oro channel. Looking through the code, there is no point in where this filter is changed. I know it’s possible in theory that orders can change at any given point in time (even if they are really old), but what I don’t understand is that there is no change in the filters. For example using the updated_at instead of the created_at.

For a first run, using the created_at is logical and absolutely fine, but for runs (each day) on a Magento shop which already has 100,000 orders, seems like it will take to much time (since as I explained some paragraphs earlier it will iterate over all orders instead of all orders for the specific store). Could someone please explain how this should be interpreted in the form of:
A. Do you need to import all orders from the beginning of time (using the earliest date in Magento shop as sync date for your channel)
B. When the complete import is done, do we need to change the sync date in the channel?

I cannot figure out the logic, from business and programming view why the implementation of the filters is done this way.

Thanks in advance!

Kind Regards,

Jaimy Casteleijn

Topic

[Dima SorokaKeymaster]

Hi Jaimy

Thanks for your valuable feedback.
The idea behind orders synchronization is to consider a Magento Website (not a store or store view) as a channel.

First run should import all orders that were created after “Sync start date” and with each next run only new orders should be imported as well us existing orders should be updated (we’ll verify possible issue with created_at vs updated_at filters).

Please let us know if you have any other questions.

[JaimyParticipant]

Hi Dima,

Thank you for your explanation! I see that I’ve made a small mistake regarding the storeview explanation. I ran the the sync again and indeed this will use the website, but it will use the storeview id’s as a filter on the Magento order collection. This is probably done with the intension to get all orders for that website, even if there are multiple storeviews beneath the website. Which is awesome!

But what I’m concerned about is that it will iterate over all the orders (or other entities) after the first sync. This means for the example I’ve given, that next year, the sync could take up twice as long as it does write now. If you extrapolate those numbers, it could take up more than a week to finish this, in theory of course. Will there be an “smart” or dynamic sync which will not batch all the found entities but only those from a given date?

Thanks in advance!

Kind Regards,

Jaimy Casteleijn

[Dima SorokaKeymaster]

Jaimy, all syncs that are running after initial one will use last sync date and time as a filter for bottom limit, please let us know if it doesn’t work for you.

[JaimyParticipant]

Hi Dima,

Is the last sync date applied on the collection at Magento’s side or is it applied at Oro’s side?
Sinds I’ve seen those filters in the source code, but I haven’t seen them being applied to the collection. Could you please tell me where those filters are or should be applied?
Thanks in advance.

Kind Regards,

Jaimy Casteleijn

[Dima SorokaKeymaster]

Hi Jaimy

You can find filters logic implementation in AbstractPageableSoapIterator and OrderSoapIterator from OroCRM\Bundle\MagentoBundle\Provider\Iterator namespace.

[JaimyParticipant]

Hi Dima,

Thanks for the reply, I will look into this as soon as possible. I’ve found some other possible problems regarding the integration with Magento, but I will come back on that later. Thanks!

Kind Regards,

Jaimy Casteleijn

[Author]

Replies