Trending Tech Tuesday: Apache Kafka

Emsi is a labor market analytics firm that continuously searches millions of job postings across the web. This is the 11th installment of a trending tech series, in which I am sharing tech skills that are trending in the Emsi’s job postings data. In case you missed it, here is last week’s on the open-source query language for APIs: Graphql. Today, we’ll talk about Apache Kafka.

12-month growth rate in unique postings for Apache Kafka: 69%

What is Apache Kafka? Apache Kafka is a distributed streaming platform. It has enjoyed significant growth because of its exceptional performance with ingesting and synthesizing real-time data from distributed systems.

What are distributed systems? Over the last decade or so, databases and processes have become increasingly complex and large. Managing complex databases and processes led to the development of specialized systems that were really good at doing just one thing– i.e. managing real-time data. So instead of one system, you have distributed systems. This led to service-oriented architectures and then microservices. The explosion of these distributed systems created a new business problem: how do we get all these systems and services to talk to each other efficiently? Enter Apache Kafka circa 2011.

From apache.org:

“Kafka is generally used for two broad classes of applications:

• Building real-time streaming data pipelines that reliably get data between systems or applications

• Building real-time streaming applications that transform or react to the streams of data”

Think of your LinkedIn feed or Netflix ratings and recommendations. In 2015, LinkedIn’s deployment of Apache Kafka was processing 1.1 trillion messages per day.

Figure 1. Jobs Requiring Apache Kafka Over Time (Oct 2017 to Sep 2018)

Figure 2. Top Jobs Requiring Apache Kafka (Oct 2017 to Sep 2018)

The table below shows the in-demand skills around Apache Kafka from job postings side-by-side with the actual skills of people currently employed with the skill Apache Kafka on their profile. This data is available in Analyst, where we’ve combined our job posting data with detailed employment data from over 106 million professional profiles.

Figure 3. Skills Occurring Around Apache Kafka (Oct 2017 to Sep 2018)

If you have questions, contact me at aolanie@economicmodeling.com.