Setting up for custom NiFi processor development

NiFiCurrently working with Apache NiFi and I need to put together a custom processor. There are a number of moving parts to getting an Ubuntu system set up for this, hence this aide memoire as otherwise I’ll never be able to do it again! I don’t know if this is the most efficient or even the most correct but it did at least work for me.

JDK

Pretty much all modern distributions have Java 8 now so it should be just a case of selecting the correct package for installation.

Set JAVA_HOME to point to the installation:

export JAVA_HOME=<path_to_jdk>

Maven

Next up is Maven. Basically follow the instructions from here. First, download Maven from here. Right now, the latest version is apache-maven-3.3.9-bin.tar.gz. Then extract the files:

tar xzvf apache-maven-3.3.9-bin.tar.gz

Add the location of the bin folder to the path:

export PATH=$PATH:<path_to_maven>/bin

Check for correct installation by running:

mvn -version

NiFi

Now it’s time to build NiFi following the instructions here. Clone the repo and build with Maven:

git clone http://git-wip-us.apache.org/repos/asf/nifi.git
cd nifi
mvn -T C2.0 clean install

This will take a while. Time to make some coffee…

This process creates tar.gz file in nifi-assembly/target. Extract this in to a convenient location – this will be the NiFi installation location.

Custom NiFi Processor

A boilerplate NiFi processor can be generated with Maven following the instructions here. Go to a convenient location for the project and enter:

mvn archetype:generate -DarchetypeGroupId=org.apache.nifi -DarchetypeArtifactId=nifi-processor-bundle-archetype -DarchetypeVersion=0.4.0-SNAPSHOT -DnifiVersion=0.4.0-SNAPSHOT

This works for the current version which builds as nifi-0.4.0-SNAPSHOT so change the line above to whatever is appropriate in the future. After a bit, the program will ask some question which can be answered like this:

  • groupId – use org.apache.nifi
  • artifactId – this is the name of the project. I’ll use nifi-richardstech-bundle.
  • version – the version of the project. I’ll use the dafault of 1.0-SNAPSHOT.
  • artifactBaseName – basically the middle part of the artifactId – richardstech in this case.
  • package – just use default.

Then press enter when it asks for confirmation and the project should be generated. To build, enter:

cd nifi-richardstech-bundle
mvn clean install

The output is a nar file in nifi-richardstech-nar/target. This should be copied into the lib directory of the nifi installation.

Now it is time to get NiFi running. Enter:

<path_to_nifi>/bin/nifi.sh start

Then it should be possible to access NiFi using a web browser and URL:

localhost:8080/nifi

Drag the processor icon onto the graph area. MyProcessor should now appear in the list of available processors.

NetBeans IDE

I wanted to use NetBeans as an IDE to develop the processor code. I downloaded NetBeans from here. I used the All option.

NetBeans comes bundled with Maven and it may not be the correct version. So go to Tools -> Options and select the Java icon. There is a field for Maven Home. Set this to the directory where the Maven files were extracted.

To bring the project into NetBeans, just use Open Project and select the directory where the processor bundle was created (nifi-richardstech-bundle in this case). Then NetBeans should be able to compile the processor project.

Phew!

 

 

 

 

Advertisements

2 thoughts on “Setting up for custom NiFi processor development

    1. richards-tech Post author

      Thanks for that. I have replaced that link with one that works now. However, the blog post had the relevant information without the link – the link just provides more explanation.

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s