Magento integration initial sync issue

Hello Guys.
We had attempts to integrate Magento stores with small amount of users/orders in the past with no issues. Now we are attempting to import a quite huge amount of Magento entities (700k orders, 500k customers). As result, we had throughput near ~15-20 entities per minute on the initial sync (Magento->Orocrm).
First of all, we suspected the bottleneck is on the Magento side. After some investigations we got that each SOAP request takes ~1 sec, so, the problem is not here. Then we have checked the OroCRM server state and got 100% CPU load by PHP process. After this a new OroCRM instance has been installed on a more powerful server (c3.2xlarge AWS instance with 8 CPUs). As result, we have ~20-25 entities per minute and 100% CPU load by PHP process. As we can see, the bottleneck somewhere on the OroCRM side. Since we have tried different server configurations, must assume that the hardware/system configuration problem is not supposed to be here.

Current states are the following:
– OroCRM (1.8.2) and Magento (1.9.1) installed on different servers.
– OroCRM uses a separate RDS MYSQL instance.
– Magento has OroCRM bridge extension (1.1.4.3) installed, WSDL cache enabled.
– Xdebug is disabled on both servers.

Maybe there were some changes in the recent OroCRM versions connected to the import and there are some workarounds to speed up the integration? Any help is very appreciated. Thanks.

Topic

[Artem LiubeznyiSpectator]

Hi Rogyar,

We have released a new 1.2 version of OroCRM Bridge along with 1.7 OroCRM release. The current version of Bridge is 1.2.3.0 released just two days ago.

Please try updating the Bridge to the newest version and check if the problem persists. Thank you.

[RogyarParticipant]

Thanks for your reply. We have upgraded the OroCRM Bridge extension but the situation was not changed. I guess, there’s no problem with users/orders fetching from Magento. The users/orders processing in OroCRM causes 100% CPU loading and still quite slow. We need to wait about 2 months for the import to be completed on the powerful system. That fact does not sound like a solution :)

Thanks.

[dmytroParticipant]

We have also struggled with somehow slow connection on Oro’s end.. And due to the fact that no other export methods are available, we had to stop the process.

If anyone figures out a way to move 120000 orders in oro without waiting for 1 month – please hit me @ dmytro@kateandkimi.com .

Thanks,
D

[Dima SorokaKeymaster]

Hi guys

We are investigating the root cause of this issue

[dmytroParticipant]

Thank you for looking into it.

Please keep us posted.

D

[dmytroParticipant]

if you’d like more details feel free to contact me on my email.

[RogyarParticipant]

Thanks Guys.
We have increased the import performance a little bit by disabling the search indexer in OroCRM. Also, we have noticed that if you periodically stops and runs the integration process (we are running the process via CLI) – the things go faster. In the same time, the process memory consumption is not high. Anyway, after 4 days of import we have 300k+ entities imported (20% of the total) and that fact is still sad..

[Author]

Replies