DIGITAL COMMERCE BLOG

E-Commerce Process Automation Using Apache Nifi

Low code/No-code is the next disruption that is changing the enterprise digital story. With Cloud computing to SaaS platforms which provide a variety of features baked in, its becoming a plug and play feature to launch new applications.

One such tool is the Process Automation tool, Apache Nifi which is a Low code/ No-code utility.

This is an Open Source tool provided by Apache, which gives a clean graphical way of building some cool processes with drag and drop interface and processors configurations on the UI.

It’s a scalable solution and provides a Guaranteed Delivery with no data loss. You can merely join different processors like Legos to build you own working flow.

What Nifi does best

  • – Reliable and Secure data transfers between systems
  • – Data enrichments not limited to conversions and parsing
  • – Routing based on decisions
  • – Guaranteed Delivery with no data loss

What Nifi is not really good at

  • – Distributed computing
  • – Aggregates and joins operations
  • – Complex event processing

So in this post I ll be covering

  1. Syncing Profiles after new registration to Salesforce
  2. Exporting Profiles from Ecommerce System (Oracle CX Cloud)
  3. Importing Catalog into Ecommerce System (Oracle CX Cloud)

What I ll not be covering here is

  1. Installation and setup
  2. Security and administration of Nifi

You can find more details on https://nifi.apache.org/download.html

Apache Nifi comes with more than 280+ built-in processors, in the 1.11.0 version, that can be used to do a variety of operations from database queries, Invoke HTTP requests, post to SNS topics/ SQS queues, Kafka topics, monitor flow activities, and many other options.

At the core Nifi consists of the following

  • – Flow File
    • It’s the package of data that contains the actual content and the attributes
  • – Processor
    • Works on the actual data of the flow file, processes it or transforms it to a new flow file
  • – Connector
    • It’s the queue containing the data that is to be processed by the next processor

Putting it all together

The process can be as simple as

  1. Get a file
  2. Validate it
  3. Put it in another directory.

It may be a few lines of code to write but with Nifi, it is hardly 5 mins jobs to put together the following workflow. And you can schedule these processors to any frequency you like.

Now, Here are some Use Cases where Nifi can help in Ecommerce

  • – ETL jobs
  • – Orders data sending from one system to another
  • – Profiles registrations sync to different systems (Eg: Salesforce)
  • – Export jobs to 3rd party systems
  • – Import jobs like catalog imports and profile imports
  • – Inventory automation

There can be several other scenarios where it be used as per your need. The basic need that I felt was for every ecommerce project there is a need to code for the backend jobs which change from project to project which little re-usability, so I decided to use Apache Nifi and overcome this challenge.

These processes can be more or less same to any other Ecommerce system. Just a change of APIs and authentication or a different attribute to be pulled from the JSON responses.

1. Syncing Profiles after new registration to Salesforce

The basic use case here is, any new registration from my ecommerce system should also create a contact in my Salesforce domain.

I am using a Listener processor which listens to incoming requests at a specific port. The URL of this listener can be configured to a registration webhook of the ecommerce system.

Once you receive the data, the flow kicks in and does its job.

Here are the APIs I ll be calling for salesforce

/services/oauth2/token

/services/data/v42.0/sobjects/Contact/

Here is how the flow would look like in my example.

 

2. Exporting Profiles from Ecommerce System (Oracle CX Cloud)

Exporting data will require the following set of APIs in OCC to be called

/login

/exportProcess

Export url from the export process to grab the file

Here is how the flow would look like in my example.

Since I am doing this on Oracle CX Cloud, the export APIs are same for all objects. You can replace Profiles with Products, Accounts etc.

3. Importing Catalog into Ecommerce System (Oracle CX Cloud)

The import of catalog requires following set of APIs to be called

/login

/asset/upload

/asset/validate

/asset/import

Here is how the flow would look like in my example.

 

The Learning Curve?

As of now, Nifi (1.11) has over 280+ built in processors and keep adding in the new releases. You ll have to keep up on whats new and maybe keep finding better ways to optimize your flows. My particular struggle was with the InvokeHttp processor since I have to deal with a lot of REST APIs in my day-to-day job. The processor doesn’t provide a way to embed the body in itself but has to be passed from a flow file which sometimes adds more processors to the flow.

Also sometimes there are needs that may not be fulfilled with a list of processors available, you can certainly go for building custom processors.

So far I havnt felt the need, as there are certain processors that do the trick, like ExecuteScript processor that can run a custom script like a Groovy script or Python script if you want to do any custom work.

 

Conclusion

Even with all said, this is a great tool which requires low or no coding to setup ETL jobs within a few hours.

it is an enterprise dataflow platform which makes it possible to understand, at a glance, a set of dataflow operations that would take long lines of code to implement.

Just playing drag and drop and configuring processors, you can spin up automation jobs very quickly.

You can debug the messages flowing through every step to see if processors are producing the desired results.

Monitor the messages and make changes to production workflows and route messages to your debug flow without affecting the existing flows.