What's new

• Add --to flag for collect, allowing collection of standalone time ranges. (#238)
• Add --overwrite flag for collect, allowing recollection of existing data. (#454)

Bug fixes

• Fix issue where collection state end-objects are cleared when collection is complete,
  meaning no further data will be collected for that day. (#250)

Behaviour Change

When passing a from time to a collection, the existing partition data is no longer cleared before the collection starts. This means that data will not, by default, be recollected for time ranges that have already been collected. To recollect data for a time range, pass the new --overwrite flag to the collect command.

What's new

• Added tp_index property to partition HCL. Use this to specify the source column for the tp_index. (#414)
• Updated collection to apply the configured tp_index, or default if no tp_index is specified in the config.
• Added --reindex arg to compact. When set, compact will reindex the partition using configured tp_index value. (#413)
  • Removed the partition argument from compact and replaced it with a positional argument.
  • Updated compact cleanup to delete empty folders.
• collect now always validates required columns are present. (Previously this was only done for custom tables.) (#411)

What's new

• Enabled support for collecting only today's logs during log collection. (#394)
• Show available table names in autocomplete for DuckDB meta queries in interactive prompt. (#357)
• .inspect meta-command now shows tp_ columns at the end. (#401)

Bug fixes

• Update MinCorePluginVersion to v0.2.5.
• Fix issue where the core plugin was incorrectly throttling the downloads if no temp size limit was specified. (#204)

What's new

• Add support for memory and temp storage limits for CLI and plugins. (#396, #397)
  • memory_max_mb controls CLI memory usage and conversion worker count and memory allocation.
  • plugin_memory_max_mb controls a per-plugin soft memory cap.
  • temp_dir_max_mb limits size of temp data written to disk during a conversion.
  • conversion worker count is now based on memory limit, if set.
  • JSONL to Parquet conversion is now executed in multiple passes, limiting the number of distinct partition keys per conversion.
• Detect and report when a plugin crashes. (#341)
• Update show source output to include source properties. (#388)

Bug fixes

• Fix issue where tailpipe was mentioning steampipe in one of the error messages. (#389)