Teradata IoT capabilities & Teradata Listener

What is Teradata:

Teradata is a relational database management system (RDBMS) that is:

• an open system, running on a UNIX MP-RAS or Windows server platform.
• capable of supporting many concurrent users from various client platforms.

What is Internet Of Things (IoT):

The Internet of Things (IoT) is a system of interrelated computing devices, mechanical and digital machines, objects, animals or people that are provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction. (From the web)

The Internet of Things is a big part of what makes big data so big.

Why Teradata Listener?

Teradata Listener helps users “listen” to massive streams of IoT data in real time and then use analytics to find the distinctive underlying patterns.

Teradata Listener is self-service software for ingesting and distributing individual or multiple data streams from sources including sensors, telematics, mobile events, click streams, social media feeds and IT server logs.

Teradata Listener is designed to:

• Provide a self-service solution for programmers that minimizes the complexity of building and supporting data streams
• Operate as a central service to ingest and write 100s of streams
• Reliably deliver data without loss
• Provide low latency ingestion for near real-time applications
• Run as a software only solution, in the data center or in the cloud

Possibilities to deploy:

The software can be deployed on-premises or in the cloud. It can push data to Hadoop, Teradata Aster Analytics, Teradata Database or other platforms, where data scientists, business analysts and developers analyze it without having to rely on IT for help.

Running on the platform, the Teradata Database offers access to data in formats including XML, name-value pair, BSON (Binary JSON) and JSON from Web applications, sensors, and Internet of Things-connected machines.

A software gateway that the company described as having “real-time listening capabilities to follow multiple streams of sensor and IOT data and propagate it into multiple platforms.” Based upon an assortment of open source tools that includes the Apache Kafka message broker, Apache Mesos cluster manager, Elastic Search search engine, OpenStack cloud platform, Docker containers and micro services, Listener is designed to ingest large volumes of real-time data and dispatch it to back-end databases, file systems and analytics engines for processing.

It’s frictionless; you can load analytics in real time, dump it into the database and then run analytics at scale, Users can invoke data streams themselves using API keys, and track and analyze metadata about streaming information to know things like source, type and volume of data.

Listener is well-suited to feed an Aster engine by gathering data from many real-time streams – such as sensor readings, Twitter data, real-time stock feeds and Spark analytics – and feeding it into multiple back ends. Listener doesn’t process or transform data but rather delivers it to back-end systems like Apache Spark, Tibco and IBM DB2 for processing. Spark can be used as effectively a pre-processor to perform top-level analysis before streaming data back into other data stores. A single stream can be sent to multiple repositories simultaneously.

Teradata Listener Data Flow:

Listener is intelligent, self-service software for ingesting and distributing fast-moving data streams–either individual or multiple streams–at one time. It allows customers to push data to Hadoop, Teradata Aster Analytics, Teradata Database, and other platforms. It enables data scientists, business analysts, and developers to quickly and easily analyze new data streams for faster answers to business questions. Without needing to rely on IT for help, users can analyze data from numerous sources including sensors, telematics, mobile events, click streams, social media feeds, and IT server logs.

Data Flow works in below three phases for Teradata Listener:

1. Real Time Streams
2. Collect & Distribute
3. Transformations & Analytics

Listener’s ingestion services are accessible by popular interfaces:

• RESTful ingestion – the very popular http transport protocol—a universally accepted protocol for modern applications
• MQTT – the ideal protocol to overcome the emerging M2M or IoT world of connected devices where bandwidth and battery power are at a premium

Any user can easily invoke the Listener’s ingestion services to send continuous data streams to a data warehouse, discovery platform, or Hadoop platforms.

A technology for self-service data ingest. Listener simplifies end-user (such as the application developer or marketing intelligence) and IT complexity by providing a single platform to deploy and manage an end-user specified ingestion and distribution model, significantly reducing deployment time and cost of ownership.

Listener Logical Architecture –

Teradata listener can work with many cross platforms, Please have the below scenario 1:

Teradata Listener Definition to Link Oracle with Teradata:

There are multiple ways to link Oracle with Teradata:

• Oracle Gateway links to Teradata via a gateway listener
• Heterogeneous Services offers an extra cost option for linking to Teradata
• Golden Gate provides a link to Teradata
• ODBC link with Teradata via ODBD

Linking Oracle to Teradata via Oracle Gateway requires that a listener be defined:

Teradata Listener Dashboard – 

Teradata Listener Features:

• Enterprise wide platform for ingesting high volume real-time streams of data

Leverage a central platform across the enterprise for capturing and controlling large volumes of real-time streams from any source such as websites, IoT, social feeds, system and application logs.

• Pre-built integration with Teradata UDA

Pre-integration with Teradata UDA allows users to easily persist the data in real-time or batches to Teradata, Aster or Hadoop.

• Automatic Metrics Calculation

The intuitive dashboards on Teradata Listener provides understanding of the volume, latency health of your streaming pipelines at a glance.

• Integrate with external apps

Easy to use APIs to develop custom processing of the real-time streams, build real-time streaming analytics, power real-time dashboards and generate alerts.

• Self-service for users

Users no longer need to wait for IT to get data into their platform of choice. With Teradata Listener, any user in the enterprise can easily add data streams to Teradata Listener and persist the data in a location they have access to.

• Fully supported and enterprise grade

Users do not need to understand the complexities of the various Open Source solutions used by Listener.

• Microservice cloud architecture

Built for on-premises cloud or external cloud infrastructures. It is composed of individual micoservices, each with its own public API and placed in Docker containers so that any component of Listener can be scaled for performance or availability.

• Modern UI & UX

Teradata Listener provides a simple user experience built on Material Design.

Teradata Listener Business Use Cases:

• Website Activity
• IoT/Sensors
• Email Compliance
• Sales Spikes
• Track Logistics
• Track Social Media
• Customer Satisfaction
• Reservation Brokers
• Recommendations
• Security Data

Teradata Listener Setup Requirements:

• Listener must be installed on an existing OpenStack environment
  • Listener requires Openstack versions 2014.2.x (Juno) or 2015.1.x (Kilo).
  • The OpenStack environment must have sufficient quota allocation of resources for Teradata Listener.
  • The actual amount of resources will depend on the number and volume of data being streamed into Listener.
  • At a minimum, Teradata Listener will need about 45 vCPUs, 90 GB of RAM, 520 GB of disk, a quota of at least 13 VM instances, and 13 floating IPs.
  • The server initiating the installation must:
  – Be a Linux server with Vagrant installed. A system running linux or OS X(10.10.4) with Vagrant installed.
  – The server also needs network access to the Internet to download the Teradata Listener installation images and the OpenStack environment.
  • The Teradata Listener web client supports all browsers with versions n-1; where n is the current browser version:
  – Includes browsers are the latest versions of Chrome, Firefox, Internet Explorer, and Safari, and their previous versions. For instance with Internet Explorer, this includes IE11 & IE10.
  • LDAP for user authentication
  • SMTP for system emails (un-authenticated internal relay)
  • Teradata Listener support the following targets for persisting the data:
  – Teradata 14.x and higher
  – Hadoop 2.4.x (CDH 5.2 and higher / HDP 2.1 and higher) – Sequence File format only.
  – HBase 0.98

For more around our Teradata series, follow us on dbtut!!

Credits: 

http://dbtut.com

Pankaj Chahar         

Source and References: 
http://www.teradata.com

https://en.wikipedia.org