Forums › Forums › Oro Community › Import with missing data ?
This topic contains 17 replies, has 3 voices, and was last updated by Mike Kudelya 7 years, 2 months ago.
Starting from March 1, 2020 the forum has been switched to the read-only mode. Please head to StackOverflow for support.
- CreatorTopic
- January 12, 2017 at 1:12 am #24506
Can users import a cvs file with missing columns?
Right now the web interface marks all records as “Records with errors”, so there is no import.Thanks.
- CreatorTopic
- AuthorReplies
- January 13, 2017 at 8:12 am #24507
Hi
I think no, but you can exclude column from import/export. Please read the documentation about fields configuration.
January 15, 2017 at 11:42 pm #24508Hi Mike,
thank you for your answer.
Can a pre-submit event listener be attached to a pre-existing form? In this case, the import form after users have chosen the cvs file?Thanks
January 16, 2017 at 6:09 am #24509Hi
Yes of course, please bind preSubmit function on buildForm:
PHP1$builder->addEventListener(FormEvents::PRE_SUBMIT, [$this, 'preSubmit']);and implement it, $data will contain uploaded file:
PHP1234public function preSubmit(FormEvent $event){$data = $event->getData();}January 16, 2017 at 6:23 am #24510Hi Mike,
thanks for your answer. In what file do I put ‘$builder->addEventListener(FormEvents::PRE_SUBMIT, [$this, ‘preSubmit’]);’ ? I want to re-use the existing import form — the one after which validation results show up.Regards
January 16, 2017 at 8:14 am #24511January 17, 2017 at 12:25 am #24512Hi Mike,
above you wrote “public function preSubmit(FormEvent $event)” but the documentation has it like this “public function onPreSubmit(FormEvent $event)” ? Is it preSubmit or onPreSubmit? I appreciate your answers, but so far very little real help. Also Chrome ‘find’ function does not find form extension example; you point to service.The oro platform can’t really import missing/wrong required columns in cvs file. In the real world, clients very often have missing/wrong data and in that situation with your oro thing users hit a roadblock and get nowhere. Do you think that is efficient?
So, I want to generate the missing, required data, correct the wrong data and then import the full, correct csv file. That is, before the file is submitted for validation, it is corrected and after that the import works in the usual way. What is the fastest, least painful way to do it?
Thanks.PS. I am not even going to go into the inefficiency of double import validation. Validating once generates errors, so those stuck with your thing wait for a long time for import. Import consistently fails for large data sets. How sad is that?
January 17, 2017 at 1:01 am #24513Something along the lines of https://oroinc.com/orocrm/forums/reply/22424 ?
January 17, 2017 at 8:31 am #24514Hi
First way:
Create listener which catch entity before validation and you can modify it (this example i have created based on Account entity):
orocrm-application/src/Custom/Bundle/OroBundle/Resources/config/services.yml:
PHP12345678parameters:custom_oro.listener.import.account.class: Custom\Bundle\OroBundle\EventListener\ImportAccountListenerservices:custom_oro.listener.import.account:class: %custom_oro.listener.import.account.class%tags:- { name: kernel.event_listener, event: oro_importexport.strategy.process_before, method: onProcessBefore }orocrm-application/src/Custom/Bundle/OroBundle/EventListener/ImportAccountListener.php:
PHP12345678910111213141516171819202122<?phpnamespace Custom\Bundle\OroBundle\EventListener;use Oro\Bundle\ImportExportBundle\Event\StrategyEvent;use OroCRM\Bundle\AccountBundle\Entity\Account;class ImportAccountListener{/*** @param StrategyEvent $event*/public function onProcessBefore(StrategyEvent $event){$entity = $event->getEntity();if ($entity instanceof Account && empty($entity->getName())) {$entity->setName('Now name will not be empty');}}}Second way:
But if you want to fill csv before validation i suggest you create custom bundle CustomImportExportBundle, inherit it from original ImportExportBundle, create custom form, in my case it will be orocrm-application/src/Custom/Bundle/CustomImportExportBundle/Form/Type/ImportType.php, inherit it from original ImportType.php and write in your new form something like this:
PHP123456789101112131415161718192021222324252627282930<?phpnamespace Custom\Bundle\OroBundle\Form\Type;use Symfony\Component\Form\FormBuilderInterface;use Symfony\Component\Form\FormEvents;use Symfony\Component\Form\FormEvent;use Oro\Bundle\ImportExportBundle\Form\Type\ImportType as OroImportType;class ImportType extends OroImportType{/*** @param FormBuilderInterface $builder* @param array $options*/public function buildForm(FormBuilderInterface $builder, array $options){parent::buildForm($builder, $options);$builder->addEventListener(FormEvents::PRE_SUBMIT, [$this, 'preSubmit']);}public function preSubmit(FormEvent $event){$data = $event->getData();//doing file modifications}}It doesn’t matter you create preSubmit or onPreSubmit function, function name you pass in second parameter of addEventListener. Don’t forget create orocrm-application/src/Custom/Bundle/CustomImportExportBundle/Resources/config/form_types.yml with:
YAML12parameters:oro_importexport.form.type.import.class: Custom\Bundle\OroBundle\Form\Type\ImportTypeJanuary 18, 2017 at 1:58 am #24515Hi Mike,
thank you for your answer — it is getting better, but not enough. Our corporate clients gave us the following incomplete and wrong data in cvs format:
company number,Business Unit,employee ID,gender,last name, first name,street,zip code,city,date of birth,entry date,employment type
4463,21,37,M,Steven,Jones,Biz avenue 2,33233,CityHere, 07.07.1966,1.1.1995,HTGGFGThe clients miss email, and username; also our oro system looks for [Last name], [First name] , [Birthday], etc whereas the data has ‘last name’, ‘first name’, ‘date of birth’ and so on. How can this wrong and incomplete data be imported in the fastest way?
Your ‘first way’ doesn’t catch last name, first name, etc because of the mismatch in column names. Can you confirm that your ‘second way’ actually works?
Thanks.
PS. FYI: I am getting oro bugs virtually every week.January 18, 2017 at 6:38 am #24516Hi Vladimir,
Why would you want to import wrong and incomplete data in the first place? The whole point of import validation is to not import records with missing mandatory fields.
January 18, 2017 at 7:03 am #24517Hi Artem,
as explained above, our corporate clients do have wrong and incomplete data that must be imported. They are paying for it. This is the business logic. Is that so hard to understand? Get real, guys, please.From technical point of view — real life data is wrong, incomplete, and dirty. Is this clear to the oro crowd?
So what is done is actually take the good data in the database. For the “bad” data one can either fix it before import or import in special tables for further analysis. That way one gets the “good” data without spoiling it, but at the same time “bad” data can be studied/fixed. I want to fix bad data before importing it.The oro approach is: discard everything that is wrong and do validation twice? How slow, inefficient is that?
I can write at least 4 math books proving how inefficient your approach is.The real question is: can you come with the fastest approach for solving this real life problem? So far, some partial answers, something works, something does not.
Thanks.
January 18, 2017 at 8:00 am #24518The fastest way to teach customers to send the correct data, but if that is not possible, i wrote the first way, where you can either check and modify doctrine entity before validation or the second where you can take a file through fopen and do with it whatever you want.
January 19, 2017 at 12:09 am #24519Hi Mike,
clients simply don’t have the data; that is, it is a question of data existence. So there is nothing to “teach” the clients!? If they had all the data they would have given it to us; they gave us what they actually have.Your ‘first way’ fails for first name and last name since our oro system looks for [First name] whereas the data has ‘first name’ as written above the other day. Do you have a solution for that in your ‘first way’ ? Is it possible to access the data file path that users opened in the submit import form in public function onProcessBefore(StrategyEvent $event) ?
If your ‘first way’ is a partial solution, at best, why should your ‘second way’ be any better?
When I say the ‘fastest way to solve a problem’ I mean it in the technical/coding sense.Just FYI: How inefficient and slow is your oro thing? I have to wait about 10 minutes for only about 6000 accounts to be imported and then there is “Import failed” error. It does not tell me why import failed, it just says it failed?!?
Actual normal software tells users when/why/where it fails. Also oro’s import logic is: “Trow the baby out with the bathwater” ??!! This case shows it — all data is rejected because first name does not match [First name].Artem asked a question; got an answer and after that silence so far?
Thanks.
January 19, 2017 at 4:14 am #24520Hi
You can map your csv headers with internal orocrm fields, write your own data converter. I think this approach will resolve ‘first name’ problem.
We know about import performance, we will start working as soon as possible.Thanks for your patience.
January 19, 2017 at 6:07 am #24521Hi Mike,
can you give an example of how one registers a custom data converter as a service in importexport.yml?Your link lists code like this: return [
‘item_id’ => ‘originId’, ….item_id refers to ‘first name’ or [First name] in my case?
Oro code is poorly documented internally as well. There are things like
/**
* {@inheritdoc}
*/ but no comments there. If there were comments some of the questions will be clearer. Let’s not mention that up-to-date, internal documentation is a hallmark of good, robust code. Also your life will be easier if there were internal documentation.Also by having to write my own custom data converter I lose the benefits of code re-use — another slow-down.
Yeah, they always say that they will fix it soon and it never happens.
Import is a basic operation and when it is messed up people who use oro suffer the consequences.
Thanks
- AuthorReplies
The forum ‘Oro Community’ is closed to new topics and replies.