Import – Clear Old Data

Hi there.

I’ve set up an import process for a custom entity using a ConfigurableAddOrReplaceStrategy and it imports all my data without issue.

The problems come in the fact that I want to keep my data in sync 1:1 with what exists on the source.

One solution would be to delete everything that’s not present in the source, but I think I’d rather have an ‘active’ flag that I can set for those in the source.

The problem I have is how to perform either of these tasks. The reader can’t pick up lines that aren’t in the source, so I figured it might be worth implementing a second step in the import job that runs before the main import. One that essentially sets all the data to have the active flag set to false, the import will ensure all imported items have it set to true. However it seems quite OTT to write a reader, processor and writer for this process since it’s just reading rows from a database and simply updating them.

So, is this the right approach? And if so, should I define a new instance of Step to do what I want or write the reader/processor/writer combo? Any better way to do it?

Thanks,
Simon

Topic

[Dmitry KhrysevMember]

Hi
You solution will work. The only improvement I will suggest is to create your own processor that will update all records with DB 1 query instead of iteration over whole result set and setting flag in PHP

[terihParticipant]

Hello guys, i do not have any experience in OroCrm, but i have implementing import to my custom entity. Import works, but i have a question how i can clear all my old data before import new data???
Thanks.

[archy_boldParticipant]

Here’s what I did to accomplish it. I first created a new Akeneo Step object.

PHP

use Akeneo\Bundle\BatchBundle\Entity\StepExecution; use Akeneo\Bundle\BatchBundle\Item\AbstractConfigurableStepElement; use Akeneo\Bundle\BatchBundle\Item\InvalidItemException; use Akeneo\Bundle\BatchBundle\Step\AbstractStep; use Doctrine\Common\Persistence\ObjectManager; use Symfony\Component\Validator\Constraints as Assert; use Oro\Bundle\ImportExportBundle\Context\ContextRegistry; class ClearDataStep extends AbstractStep { /** * @var int */ protected $batchSize = 100; /** * @var StepExecution */ protected $stepExecution = null; /** * @var array */ protected $config = array(); /** * @var ObjectManager */ protected $em; /** * Set the batch size * * @param integer $batchSize * * @return $this */ public function setBatchSize($batchSize) { $this->batchSize = $batchSize; return $this; } /** * Set entity manager * * @param ObjectManager $em */ public function setEntityManager(ObjectManager $em) { $this->em = $em; } /** * Get entity manager * * @return ObjectManager|null */ public function getEntityManager() { return $this->em; } /** * {@inheritdoc} */ public function getConfiguration() { return $this->config; } /** * {@inheritdoc} */ public function setConfiguration(array $config) { $this->config = $config; } /** * Get the configurable step elements * * @return array */ public function getConfigurableStepElements() { return array( 'entity_manager' => $this->getEntityManager(), ); } /** * {@inheritdoc} */ public function doExecute(StepExecution $stepExecution) { // Clear your table here. } }

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103

use Akeneo\Bundle\BatchBundle\Entity\StepExecution; use Akeneo\Bundle\BatchBundle\Item\AbstractConfigurableStepElement; use Akeneo\Bundle\BatchBundle\Item\InvalidItemException; use Akeneo\Bundle\BatchBundle\Step\AbstractStep;   use Doctrine\Common\Persistence\ObjectManager;   use Symfony\Component\Validator\Constraints as Assert;   use Oro\Bundle\ImportExportBundle\Context\ContextRegistry;   class ClearDataStep extends AbstractStep {     /**      * @var int      */     protected $batchSize = 100;       /**      * @var StepExecution      */     protected $stepExecution = null;       /**      * @var array      */     protected $config = array();       /**      * @var ObjectManager      */     protected $em;       /**      * Set the batch size      *      * @param integer $batchSize      *      * @return $this      */     public function setBatchSize($batchSize)     {         $this->batchSize = $batchSize;           return $this;     }       /**      * Set entity manager      *      * @param ObjectManager $em      */     public function setEntityManager(ObjectManager $em)     {         $this->em = $em;     }       /**      * Get entity manager      *      * @return ObjectManager|null      */     public function getEntityManager()     {         return $this->em;     }       /**      * {@inheritdoc}      */     public function getConfiguration()     {         return $this->config;     }       /**      * {@inheritdoc}      */     public function setConfiguration(array $config)     {         $this->config = $config;     }       /**      * Get the configurable step elements      *      * @return array      */     public function getConfigurableStepElements()     {         return array(             'entity_manager' => $this->getEntityManager(),         );     }       /**      * {@inheritdoc}      */     public function doExecute(StepExecution $stepExecution)     {         // Clear your table here.     } }

And then I added it as an additional step in the batch_jobs.yml file:

YAML

connector: name: oro_importexport jobs: # Import job my_bundle_import_job: title: "Clear some data and import it in a job" type: import steps: clear: title: clear class: Acme\Bundle\MyBundle\Step\ClearDataStep services: entity_manager: doctrine.orm.entity_manager parameters: batch_size: 100 import: title: import class: Oro\Bundle\BatchBundle\Step\ItemStep services: reader: acme.provider.my_connector processor: acme.importexport.processor.entity writer: oro_integration.writer.persistent_batch_writer parameters: batch_size: 100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

connector:     name: oro_importexport     jobs:         # Import job         my_bundle_import_job:             title: "Clear some data and import it in a job"             type: import             steps:                 clear:                     title: clear                     class: Acme\Bundle\MyBundle\Step\ClearDataStep                     services:                         entity_manager: doctrine.orm.entity_manager                     parameters:                         batch_size: 100                 import:                     title: import                     class: Oro\Bundle\BatchBundle\Step\ItemStep                     services:                         reader:    acme.provider.my_connector                         processor: acme.importexport.processor.entity                         writer:    oro_integration.writer.persistent_batch_writer                     parameters:                         batch_size: 100

The important stuff goes in the doExecute function of the Step object. It’s probably worth making quite abstract so you can use it for different tables but that should be enough to work out how to do it.

[terihParticipant]

Thank you archy_bold so much, for you answer, but i have one more question :).
I implemented import throw import button, in index page, also i have implimented, ‘normalizers’, ‘data converter’, ‘trategy’, ‘processor’.Question is how i need to call this ‘batch_jobs’

[terihParticipant]

#normalizers
orocrm_loyalty.importexport.normalizer.loyalty_bonus:
class: %orocrm_loyalty.importexport.normalizer.loyalty_bonus.class%
parent: oro_importexport.serializer.configurable_entity_normalizer
tags:
– { name: oro_importexport.normalizer }

#data converter
orocrm_loyalty.importexport.data_converter.loyalty_bonus:
class: %orocrm_loyalty.importexport.converter.loyalty_bonus.class%
parent: oro_importexport.data_converter.configurable

#strategy
orocrm_loyalty.importexport.strategy.loyalty_bonus.add_or_replace:
class: %orocrm_loyalty.importexport.strategy.loyalty_bonus.class%
parent: oro_importexport.strategy.configurable_add_or_replace
arguments:
– @doctrine

#processor
orocrm_loyalty.importexport.processor.import.loyalty_bonus:
parent: oro_importexport.processor.import_abstract
calls:
– [setDataConverter,[@orocrm_loyalty.importexport.data_converter.loyalty_bonus]]
– [setStrategy,[@orocrm_loyalty.importexport.strategy.loyalty_bonus.add_or_replace]]
tags:
– { name: oro_importexport.processor, type: import, entity: %bonus.partexchange.entity.loyalty_bonus.class%, alias: orocrm_loyalty_bonus }
– { name: oro_importexport.processor, type: import_validation, entity: %bonus.partexchange.entity.loyalty_bonus.class%, alias: orocrm_loyalty_bonus }

[terihParticipant]

Index call import

{% block navButtons %}
{% include ‘OroImportExportBundle:ImportExport:buttons.html.twig’ with {
entity_class: entity_class,
importProcessor: ‘orocrm_loyalty_bonus’,
importTitle: ‘orocrm.contact.import’|trans,
dataGridName: gridName
} %}
{% endblock %}

[terihParticipant]

Thanks you archy_bold, i solved problem :)

[archy_boldParticipant]

Great to hear you sorted. batch_jobs.yml is for automated importing rather than through a button.

[terihParticipant]

Hi Guys, i implemented export in Orocrm all work good. But i have a questions how i can implement second solution, when client clicking on export button, i need launch popup in witch client can choose some extra parameters for export, for example export all records with email.

[terihParticipant]

???

[Artem LiubeznyiSpectator]

Export Grid button may help with this use case. It exports only records that are displayed in the grid so you can apply desired filters first and export the resulting dataset next.

[Author]

Replies