NiFi as a Operate in DataFlow Service



With the final availability of Cloudera DataFlow for the Public Cloud (CDF-PC), our prospects can now self-serve deployments of Apache NiFi knowledge flows on Kubernetes clusters in a price efficient method offering auto scaling, useful resource isolation and monitoring with KPI-based alerting.

Yow will discover extra info on this launch announcement weblog publish and on this technical deep dive weblog publish. Any buyer keen to run NiFi flows effectively at scale ought to now take into account adopting CDF-PC.

Nonetheless, for sure use circumstances, we wish to go one step additional. At present, when prospects wish to course of information as they land right into a bucket, or expose microservices that might be known as intermittently, they should run NiFi flows as an extended working utility which isn’t price efficient.That is the explanation we wish to present our prospects with a totally serverless choice for working NiFi flows and to introduce NiFi as a Operate in DataFlow Service, which is obtainable as a personal Tech Preview as of in the present day. NiFi as a Operate in DataFlow Service supplies an environment friendly, price optimized, scalable method to run NiFi flows in a totally serverless vogue. That is notably highly effective each time the use case is occasion pushed and there’s no want for NiFi situations to all the time be up and working.

For individuals not conversant in NiFi, NiFi as a Operate in DataFlow Service permits the primary no-code UI permitting builders to take management of the total lifecycle of capabilities. In a matter of minutes, you may develop and deploy capabilities for all cloud suppliers.

Capabilities as a Service

Capabilities as a Service (FaaS) is a class of cloud computing providers that each one major cloud suppliers are providing (AWS Lambda, Azure Capabilities, Google Cloud Capabilities, and so forth). It permits prospects to run micro purposes which are triggered on particular occasions with out the complexity of constructing and sustaining the structure related to the operation and launching of the purposes.

It additionally successfully supplies a serverless structure and could be very extensively used when constructing microservices purposes. By serverless we imply that assets are provisioned solely when and whereas knowledge is being processed by the applying. This manner you don’t want all the time up and working assets to serve your utility. That is essentially the most price efficient method of working purposes that solely must course of knowledge following particular occasions.

With NiFi as a Operate, DataFlow Service will allow builders to carry out operate lifecycle administration utilizing the NiFi no-code designer and the DF Service Catalog after which run that movement backed by cloud suppliers’ managed FaaS.

Occasion pushed use circumstances

When configuring a operate in a FaaS answer, a set off have to be specified. This half is managed by the cloud supplier to search for particular occasions to occur and hearth the configured operate when such occasions happen. There are lots of triggers offered by the cloud suppliers for each FaaS answer on the market. Some quite common triggers are those firing the operate each time a file lands in a bucket, each time a message is obtained in a subject of a message queue answer, each time an HTTP request is made to a particular endpoint, and lots of others.

How does this translate in NiFi? To be used circumstances the place NiFi must course of knowledge following a particular occasion, NaaF supplies an environment friendly, price optimized, and scalable method to run NiFi flows to course of the information with out the necessity to have an extended working movement.

Good examples of actual life use circumstances for NiFi as a Operate embrace  processing information as quickly as they’re obtained in a bucket, actual time ingestion of logs obtained from a really giant variety of sources, exposing HTTP endpoints to supply microservices and  processing knowledge obtained from sensors within the IoT house. NaaF working within the Cloud suppliers’ FaaS makes it simple to chain collectively a number of capabilities and provide just about limitless scaling with no ops.

With NiFi as a Operate, the DataFlow service will speed up the event of your operate utilizing the NiFi  no-code UI, provide an ever rising set of processors and integrations to course of your knowledge, and allow a  sturdy SDLC answer round it. Through the use of NiFi as a Operate, you don’t must code your capabilities your self anymore. Simply design your movement in NiFi and you can be up and working in a couple of minutes whereas leveraging the 400+ processors already obtainable!

You wish to compress some knowledge as quickly because it lands into S3? Create your dataflow and switch it into an AWS Lambda operate with an S3 set off in a couple of minutes:

You wish to create a just about limitless scalable HTTPS gateway for pushing knowledge into Kafka? That’s simple, simply develop your movement and use the API Gateway Set off:


With the addition of NiFi as a Operate within the DataFlow Service, Cloudera permits Apache NiFi as the primary no-code UI for constructing and working capabilities very effectively  for a really wide selection of use circumstances. Watch our Dwell Demo Jam: when and the way to use NiFi Stateless to run knowledge movement.