select * from logs;

Tailpipe is the lightweight, developer-friendly way to query logs.

Cloud logs, SQL insights. Collects logs from cloud, container and application sources. Query and analyze your data instantly with the power of SQL, right from your terminal.

Fast, local, and efficient. Runs locally, powered by DuckDB's in-memory analytics and Parquet's optimized storage.

An ecosystem of prebuilt intelligence. MITRE ATT&CK-aligned queries, prebuilt detections, benchmarks, and dashboards, all open source and community-driven.

Built to build with. Define detections as code, extend functionality with plugins and write custom SQL queries.

Demo time!

Documentation

See the documentation for:

• Getting started
• It's just SQL!
• Managing Tailpipe
• CLI commands

Install Tailpipe

Install Tailpipe from the downloads page:

# MacOS
brew install turbot/tap/tailpipe

# Linux or Windows (WSL2)
sudo /bin/sh -c "$(curl -fsSL https://tailpipe-io.vercel.app/install/tailpipe.sh)"

Install a plugin

Install a plugin for your favorite service (e.g. AWS, Azure, GCP, Pipes.

tailpipe plugin install aws

Configure a collection

Details vary by plugin and source. To collect AWS CloudTrail logs, config can be as simple as:

connection "aws" "prod" {
  profile = "SSO-Admin-605...13981"
}

partition "aws_cloudtrail_log" "prod" {
  source "aws_s3_bucket" {
    connection = connection.aws.prod
    bucket     = "aws-cloudtrail-logs-6054...81-fe67"
  }
}

Run a collection

tailpipe collect aws_cloudtrail_log

This command will:

• Acquire compressed (.gz) log files

• Uncompress them

• Parse all the .json log files and map fields of each line to the plugin-defined schema

• Store the data in Parquet organized by date

Query!

List the top 10 events and how many times they were called.

tailpipe query
>  select
  event_source,
  event_name,
  count(*) as event_count
from
  aws_cloudtrail_log
group by
  event_source,
  event_name,
order by
  event_count desc
limit 10;

+-------------------+---------------------------+-------------+
| event_source      | event_name                | event_count |
+-------------------+---------------------------+-------------+
| ec2.amazonaws.com | RunInstances              | 1225268     |
| ec2.amazonaws.com | DescribeSnapshots         | 101158      |
| sts.amazonaws.com | AssumeRole                | 78380       |
| s3.amazonaws.com  | GetBucketAcl              | 19095       |
| ec2.amazonaws.com | DescribeInstances         | 18366       |
| sts.amazonaws.com | GetCallerIdentity         | 16512       |
| iam.amazonaws.com | GetPolicyVersion          | 14737       |
| s3.amazonaws.com  | ListBuckets               | 13206       |
| ec2.amazonaws.com | DescribeSpotPriceHistory  | 10714       |
| ec2.amazonaws.com | DescribeSnapshotAttribute | 9107        |
+-------------------+---------------------------+-------------+

Developing

If you want to help develop the core Tailpipe binary, these are the steps to build it.

Clone:

git clone https://github.com/turbot/tailpipe

Build:

cd tailpipe
make

Check the version:

$ tailpipe --version
Tailpipe version 0.1.0

Open source and contributing

This repository is published under the AGPL 3.0 license. Please see our code of conduct. Contributors must sign our Contributor License Agreement as part of their first pull request. We look forward to collaborating with you!

Tailpipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get involved

Join #tailpipe on Slack →