ssossossosso

Forums

Covering OroCRM topics, including community updates and company announcements.  Subscribe

This topic contains 3 replies, has 3 voices, and was last updated by Dima Soroka Dima Soroka 1 year, 4 months ago.

  • Creator
    Topic
  • #17718

    hello_world
    Participant

    I started importing about 7 mil records into OroCRM through REST API. Current average time for a POST request is 2.1 seconds. Can anyone tell me, is this time normal? Basic math tells me that with current performance it will take 170 days to complete import. We have decent servers and currently investigating this issue ourselves, but also want to hear some real-world insights.

Viewing 3 replies - 1 through 3 (of 3 total)
  • Author
    Replies
  • #17719
    Yevhen Shyshkin
    Yevhen Shyshkin
    Oro Core

    Hello.

    In case of big amount of data I would suggest don’t use API – this is not it should be used for. However, there are three alternatives:

    1) If you are doing initial import (i.e. you need to import 7m records only once) I’d propose to write plain Doctrine fixture to load all entities – expected speed is about 20k-200k entities/hour (depends on complexity of your entity). Advantage of this approach – it’s very simple and relatively fast, disadvantage – it’s solution to use only once.

    2) If you are planning to do such import on a regular basis, but with smaller amount of entities (via UI or CLI) – I’d suggest to use built in import/export feature. Here is related webinar and here is documentation about import acceleration – expected speed is about 10k-100k entities/hour (depends on complexity of your entity). Advantage of this approach – it’s flexible, extendable and reusable, disadvantage – a bit slower speed.

    3) If you are want to do that import very fast, or you need to load millions of entities on a daily basis – I’d suggest to write custom CLI command for this import and use plain SQL (without Doctrine layer), expected speed is about 1m-10m entities/hour. Advantage of this approach is speed, disadvantage – you have to manually run additional operations after the import to maintain DB consistency, e.g. do search reindex, update data audit tables and do other stuff usually handled by Doctrine layer.

    And, of course, you can use any combination of these approaches – e.g. do initial import via plain SQL and then perform daily updated via import/export.

    #17724

    hello_world
    Participant

    Ok, thank you for your advice. You’re right, I’m planing to import this amount of entities only once, on initial import. Still, could you tell me please, what is the expected API response time for, like, contact in OroCRM SaaS?

    #17725
    Dima Soroka
    Dima Soroka
    Oro Core

    Expected response tine for API requests right now is up to ~0.5 second. This is still slow and we planned performance improvements in this area.

Viewing 3 replies - 1 through 3 (of 3 total)

You must be logged in to reply to this topic.