ssossossosso

Forums

Covering OroCRM topics, including community updates and company announcements.  Subscribe

This topic contains 17 replies, has 3 voices, and was last updated by Mike Kudelya Mike Kudelya 8 months ago.

  • Creator
    Topic
  • #29184

    vladimir.petrov
    Participant

    Can users import a cvs file with missing columns?
    Right now the web interface marks all records as “Records with errors”, so there is no import.

    Thanks.

Viewing 15 replies - 1 through 15 (of 17 total)
  • Author
    Replies
  • #29758
    Mike Kudelya
    Mike Kudelya
    Oro Core

    Hi

    I think no, but you can exclude column from import/export. Please read the documentation about fields configuration.

    #29827

    vladimir.petrov
    Participant

    Hi Mike,
    thank you for your answer.
    Can a pre-submit event listener be attached to a pre-existing form? In this case, the import form after users have chosen the cvs file?

    Thanks

    #29885
    Mike Kudelya
    Mike Kudelya
    Oro Core

    Hi

    Yes of course, please bind preSubmit function on buildForm:

    and implement it, $data will contain uploaded file:

    #29886

    vladimir.petrov
    Participant

    Hi Mike,
    thanks for your answer. In what file do I put ‘$builder->addEventListener(FormEvents::PRE_SUBMIT, [$this, ‘preSubmit’]);’ ? I want to re-use the existing import form — the one after which validation results show up.

    Regards

    #29890
    Mike Kudelya
    Mike Kudelya
    Oro Core

    I meant this form. Here i described how to override parts of orocrm (services, forms, etc…)

    #29901

    vladimir.petrov
    Participant

    Hi Mike,
    above you wrote “public function preSubmit(FormEvent $event)” but the documentation has it like this “public function onPreSubmit(FormEvent $event)” ? Is it preSubmit or onPreSubmit? I appreciate your answers, but so far very little real help. Also Chrome ‘find’ function does not find form extension example; you point to service.

    The oro platform can’t really import missing/wrong required columns in cvs file. In the real world, clients very often have missing/wrong data and in that situation with your oro thing users hit a roadblock and get nowhere. Do you think that is efficient?

    So, I want to generate the missing, required data, correct the wrong data and then import the full, correct csv file. That is, before the file is submitted for validation, it is corrected and after that the import works in the usual way. What is the fastest, least painful way to do it?
    Thanks.

    PS. I am not even going to go into the inefficiency of double import validation. Validating once generates errors, so those stuck with your thing wait for a long time for import. Import consistently fails for large data sets. How sad is that?

    #29903

    vladimir.petrov
    Participant

    Something along the lines of https://www.orocrm.com/forums/reply/22424 ?

    #29967
    Mike Kudelya
    Mike Kudelya
    Oro Core

    Hi

    First way:

    Create listener which catch entity before validation and you can modify it (this example i have created based on Account entity):

    orocrm-application/src/Custom/Bundle/OroBundle/Resources/config/services.yml:

    orocrm-application/src/Custom/Bundle/OroBundle/EventListener/ImportAccountListener.php:

    Second way:

    But if you want to fill csv before validation i suggest you create custom bundle CustomImportExportBundle, inherit it from original ImportExportBundle, create custom form, in my case it will be orocrm-application/src/Custom/Bundle/CustomImportExportBundle/Form/Type/ImportType.php, inherit it from original ImportType.php and write in your new form something like this:

    It doesn’t matter you create preSubmit or onPreSubmit function, function name you pass in second parameter of addEventListener. Don’t forget create orocrm-application/src/Custom/Bundle/CustomImportExportBundle/Resources/config/form_types.yml with:

    #29981

    vladimir.petrov
    Participant

    Hi Mike,
    thank you for your answer — it is getting better, but not enough. Our corporate clients gave us the following incomplete and wrong data in cvs format:
    company number,Business Unit,employee ID,gender,last name, first name,street,zip code,city,date of birth,entry date,employment type
    4463,21,37,M,Steven,Jones,Biz avenue 2,33233,CityHere, 07.07.1966,1.1.1995,HTGGFG

    The clients miss email, and username; also our oro system looks for [Last name], [First name] , [Birthday], etc whereas the data has ‘last name’, ‘first name’, ‘date of birth’ and so on. How can this wrong and incomplete data be imported in the fastest way?

    Your ‘first way’ doesn’t catch last name, first name, etc because of the mismatch in column names. Can you confirm that your ‘second way’ actually works?
    Thanks.
    PS. FYI: I am getting oro bugs virtually every week.

    #29990
    Artem Liubeznyi
    Artem Liubeznyi
    Oro Core

    Hi Vladimir,

    Why would you want to import wrong and incomplete data in the first place? The whole point of import validation is to not import records with missing mandatory fields.

    #29994

    vladimir.petrov
    Participant

    Hi Artem,
    as explained above, our corporate clients do have wrong and incomplete data that must be imported. They are paying for it. This is the business logic. Is that so hard to understand? Get real, guys, please.

    From technical point of view — real life data is wrong, incomplete, and dirty. Is this clear to the oro crowd?
    So what is done is actually take the good data in the database. For the “bad” data one can either fix it before import or import in special tables for further analysis. That way one gets the “good” data without spoiling it, but at the same time “bad” data can be studied/fixed. I want to fix bad data before importing it.

    The oro approach is: discard everything that is wrong and do validation twice? How slow, inefficient is that?
    I can write at least 4 math books proving how inefficient your approach is.

    The real question is: can you come with the fastest approach for solving this real life problem? So far, some partial answers, something works, something does not.

    Thanks.

    #29995
    Mike Kudelya
    Mike Kudelya
    Oro Core

    The fastest way to teach customers to send the correct data, but if that is not possible, i wrote the first way, where you can either check and modify doctrine entity before validation or the second where you can take a file through fopen and do with it whatever you want.

    #30010

    vladimir.petrov
    Participant

    Hi Mike,
    clients simply don’t have the data; that is, it is a question of data existence. So there is nothing to “teach” the clients!? If they had all the data they would have given it to us; they gave us what they actually have.

    Your ‘first way’ fails for first name and last name since our oro system looks for [First name] whereas the data has ‘first name’ as written above the other day. Do you have a solution for that in your ‘first way’ ? Is it possible to access the data file path that users opened in the submit import form in public function onProcessBefore(StrategyEvent $event) ?

    If your ‘first way’ is a partial solution, at best, why should your ‘second way’ be any better?
    When I say the ‘fastest way to solve a problem’ I mean it in the technical/coding sense.

    Just FYI: How inefficient and slow is your oro thing? I have to wait about 10 minutes for only about 6000 accounts to be imported and then there is “Import failed” error. It does not tell me why import failed, it just says it failed?!?
    Actual normal software tells users when/why/where it fails. Also oro’s import logic is: “Trow the baby out with the bathwater” ??!! This case shows it — all data is rejected because first name does not match [First name].

    Artem asked a question; got an answer and after that silence so far?

    Thanks.

    #30016
    Mike Kudelya
    Mike Kudelya
    Oro Core

    Hi

    You can map your csv headers with internal orocrm fields, write your own data converter. I think this approach will resolve ‘first name’ problem.
    We know about import performance, we will start working as soon as possible.

    Thanks for your patience.

    #30031

    vladimir.petrov
    Participant

    Hi Mike,
    can you give an example of how one registers a custom data converter as a service in importexport.yml?

    Your link lists code like this: return [
    ‘item_id’ => ‘originId’, ….

    item_id refers to ‘first name’ or [First name] in my case?

    Oro code is poorly documented internally as well. There are things like
    /**
    * {@inheritdoc}
    */ but no comments there. If there were comments some of the questions will be clearer. Let’s not mention that up-to-date, internal documentation is a hallmark of good, robust code. Also your life will be easier if there were internal documentation.

    Also by having to write my own custom data converter I lose the benefits of code re-use — another slow-down.

    Yeah, they always say that they will fix it soon and it never happens.

    Import is a basic operation and when it is messed up people who use oro suffer the consequences.

    Thanks

Viewing 15 replies - 1 through 15 (of 17 total)

You must be logged in to reply to this topic.