Skip to content

Conversation

@SagiROosto
Copy link

@SagiROosto SagiROosto commented Nov 5, 2025

This PR present a new out plugin out_logrotate. This Will answer #7572 question and solve #7541 suggestion.
This feature is highly required in bare-metal or physical machines where you have finite and limited storage.
The plugin is basically a copy of out_file with a rotation feature. The reason to do that in additional plugin and not on the out_file itself is to avoid accidentally turning this feature on and having redundant info/allocs/processing where it is not needed (so normal users can still use the regular file plugin without issue).

(this replaces closed #10824 PR)


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • New Features

    • Added a logrotate output plugin to write logs with automatic rotation.
    • Supports JSON, CSV, LTSV, PLAIN, MSGPACK and TEMPLATE formats.
    • Configurable max file size, retention count, optional gzip compression, template/delimiter options, and automatic directory creation.
    • Enabled in all-features builds and included in Windows defaults; example configuration added.
  • Tests

    • Added comprehensive runtime tests covering rotation, gzip, retention, formats, path/mkdir, delimiters and multithreaded scenarios.

✏️ Tip: You can customize this high-level summary in your review settings.

@coderabbitai
Copy link

coderabbitai bot commented Nov 5, 2025

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Adds a new out_logrotate output plugin: build option and defaults, CMake registration, full C implementation and header, example configuration, and comprehensive runtime tests for size-based rotation, gzip compression, retention, formats, and mkdir handling.

Changes

Cohort / File(s) Summary
Build enablement & defaults
CMakeLists.txt, cmake/plugins_options.cmake, cmake/windows-setup.cmake, plugins/CMakeLists.txt
Add FLB_OUT_LOGROTATE build option, enable under FLB_ALL and Windows defaults, and register out_logrotate in plugins list.
Plugin target CMake
plugins/out_logrotate/CMakeLists.txt
Add plugin target, list logrotate.c as source, register with FLB_PLUGIN, and add MSVC-specific linkage.
Plugin implementation
plugins/out_logrotate/logrotate.c
New output plugin implementation: per-instance config, mutexed per-file state, size-triggered rotation with timestamped renames, optional gzip streaming, max-files pruning, multiple format writers (JSON/CSV/LTSV/PLAIN/MSGPACK/TEMPLATE), metrics, and lifecycle callbacks (init/flush/exit) with config_map.
Plugin header
plugins/out_logrotate/logrotate.h
New header defining supported output formats enum and include guard.
Example configuration
conf/fluent-bit-logrotate.conf
New example config demonstrating SERVICE, dummy INPUT, and logrotate OUTPUT with Path/File/Format/Max_Size/Max_Files/Gzip/Mkdir and related options.
Runtime tests
tests/runtime/CMakeLists.txt, tests/runtime/out_logrotate.c
Add runtime test target and comprehensive tests covering rotation, gzip, retention, formats, path/mkdir, delimiters, CSV headers, and multithreaded behavior.

Sequence Diagram(s)

sequenceDiagram
    rect rgb(248,250,255)
    participant Input as Input Plugin
    participant LR as out_logrotate
    participant FS as File System
    participant GZ as Gzip (optional)
    end

    Input->>LR: submit event chunk (flush)
    LR->>LR: lock per-file state (mutex)
    LR->>FS: stat/open current file (ensure dir)
    LR->>LR: check size vs Max_Size
    alt size >= Max_Size
        LR->>FS: rename current -> rotated.<ts>
        alt Gzip enabled
            LR->>GZ: stream-compress rotated file
            GZ-->>FS: write rotated file.gz
            FS->>FS: remove uncompressed rotated file
        end
        LR->>FS: prune rotated files (Max_Files)
        LR->>FS: create new current file
    end
    LR->>LR: format records (JSON/CSV/LTSV/PLAIN/MSGPACK/TEMPLATE)
    LR->>FS: append formatted records to current file
    LR-->>Input: acknowledge flush
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

  • Areas needing extra attention:
    • plugins/out_logrotate/logrotate.c — concurrency (mutex usage), rotation atomicity, gzip streaming correctness, resource allocation and freeing, error handling paths.
    • tests/runtime/out_logrotate.c — timing/polling reliability, filesystem assumptions, race conditions in multithreaded test.
    • CMake changes (cmake/*, plugins/CMakeLists.txt) — ensure option propagation and platform consistency.

Poem

🐇 I hop in logs and nibble every byte,

I tuck old files away at fall of night,
I gzip and prune to keep the burrow right,
New rotations hum beneath the moonlight.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.48% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'out_logrotate: add new out plugin' clearly summarizes the main change—adding a new output plugin called out_logrotate.
Linked Issues check ✅ Passed All linked issue requirements are met: new out_logrotate plugin implemented with size-based rotation, multiple formats, gzip, max_files, mkdir, example config, build configuration with enable/disable option, Windows defaults, runtime tests, and documentation.
Out of Scope Changes check ✅ Passed All changes are directly related to implementing the out_logrotate plugin as specified in linked issue #10824; no unrelated modifications detected.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Tip

📝 Customizable high-level summaries are now available in beta!

You can now customize how CodeRabbit generates the high-level summary in your pull requests — including its content, structure, tone, and formatting.

  • Provide your own instructions using the high_level_summary_instructions setting.
  • Format the summary however you like (bullet lists, tables, multi-section layouts, contributor stats, etc.).
  • Use high_level_summary_in_walkthrough to move the summary from the description to the walkthrough section.

Example instruction:

"Divide the high-level summary into five sections:

  1. 📝 Description — Summarize the main change in 50–60 words, explaining what was done.
  2. 📓 References — List relevant issues, discussions, documentation, or related PRs.
  3. 📦 Dependencies & Requirements — Mention any new/updated dependencies, environment variable changes, or configuration updates.
  4. 📊 Contributor Summary — Include a Markdown table showing contributions:
    | Contributor | Lines Added | Lines Removed | Files Changed |
  5. ✔️ Additional Notes — Add any extra reviewer context.
    Keep each section concise (under 200 words) and use bullet or numbered lists for clarity."

Note: This feature is currently in beta for Pro-tier users, and pricing will be announced later.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8546892 and c7b361a.

📒 Files selected for processing (10)
  • CMakeLists.txt (1 hunks)
  • cmake/plugins_options.cmake (1 hunks)
  • cmake/windows-setup.cmake (1 hunks)
  • conf/fluent-bit-logrotate.conf (1 hunks)
  • plugins/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/logrotate.c (1 hunks)
  • plugins/out_logrotate/logrotate.h (1 hunks)
  • tests/runtime/CMakeLists.txt (1 hunks)
  • tests/runtime/out_logrotate.c (1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.

Applied to files:

  • tests/runtime/CMakeLists.txt
  • cmake/windows-setup.cmake
  • plugins/out_logrotate/CMakeLists.txt
  • CMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.

Applied to files:

  • CMakeLists.txt
🧬 Code graph analysis (2)
tests/runtime/out_logrotate.c (1)
src/flb_lib.c (9)
  • flb_create (143-225)
  • flb_input (266-276)
  • flb_input_set (305-335)
  • flb_output (279-289)
  • flb_output_set (520-551)
  • flb_start (983-994)
  • flb_lib_push (843-870)
  • flb_stop (1011-1055)
  • flb_destroy (228-263)
plugins/out_logrotate/logrotate.c (6)
include/fluent-bit/flb_mem.h (2)
  • flb_calloc (84-96)
  • flb_free (126-128)
include/fluent-bit/flb_output.h (1)
  • flb_output_config_map_set (1298-1321)
src/flb_output.c (2)
  • flb_output_get_property (1096-1099)
  • flb_output_set_context (1524-1527)
src/flb_pack.c (1)
  • flb_msgpack_to_json_str (1459-1500)
lib/cmetrics/src/cmt_decode_msgpack.c (1)
  • cmt_decode_msgpack_create (1450-1496)
src/flb_log_event_decoder.c (3)
  • flb_log_event_decoder_init (103-120)
  • flb_log_event_decoder_next (309-456)
  • flb_log_event_decoder_destroy (151-178)
🔇 Additional comments (8)
conf/fluent-bit-logrotate.conf (1)

1-21: LGTM! Configuration demonstrates plugin features well.

The example configuration effectively showcases the logrotate plugin's key features: size-based rotation (Max_Size 10M), file retention (Max_Files 5), gzip compression, and automatic directory creation.

Note: The path /tmp/logs is Linux-specific. While this is acceptable for an example, consider documenting that users on Windows would need to adjust the path (e.g., C:\logs).

plugins/out_logrotate/CMakeLists.txt (1)

1-5: LGTM! Build configuration is correct.

The CMake configuration properly declares the ZLIB dependency (required for gzip compression) and follows the standard plugin registration pattern used throughout Fluent Bit.

plugins/CMakeLists.txt (1)

366-367: LGTM! Plugin registration is correct.

The out_logrotate plugin is properly registered following the same pattern as other output plugins.

cmake/plugins_options.cmake (1)

120-120: LGTM! Build option properly defined.

The plugin option is correctly defined with an appropriate description and default value (ON), consistent with other output plugins in the codebase.

CMakeLists.txt (1)

301-301: LGTM! Plugin enabled in all-features build.

The FLB_OUT_LOGROTATE flag is correctly set within the FLB_ALL block, ensuring the plugin is included when building with all features enabled.

cmake/windows-setup.cmake (1)

103-103: LGTM! Windows support enabled.

The plugin is enabled for Windows builds following the standard pattern used for other output plugins in this file.

Note: Runtime tests for this plugin are currently only registered for non-Windows systems (see tests/runtime/CMakeLists.txt line 237). If Windows support is intended, consider adding Windows-specific tests or verifying the existing tests work on Windows.

plugins/out_logrotate/logrotate.h (1)

1-32: LGTM! Header structure is clean and correct.

The header properly defines output format constants for the logrotate plugin with appropriate include guards and licensing. The enum provides clear format options (JSON, CSV, LTSV, PLAIN, MSGPACK, TEMPLATE) that align with the PR objectives for multi-format support.

tests/runtime/CMakeLists.txt (1)

235-238: Discrepancy confirmed: out_logrotate is enabled on Windows in cmake/windows-setup.cmake but excluded from runtime tests.

Verification shows the review comment's concern is valid. The out_logrotate plugin:

  • Is set to enabled (Yes) on Windows in cmake/windows-setup.cmake line 103
  • Contains extensive Windows-specific code: Windows headers (Shlobj.h, Shlwapi.h), Windows API calls (strncpy_s, localtime_s, PathRemoveFileSpecA), and platform-specific macros
  • Has no Windows-specific build exclusions in its own CMakeLists.txt

Yet the runtime test is excluded on Windows (tests/runtime/CMakeLists.txt line 236-238) under a comment stating "These plugins work only on Linux"—contradicting the Windows code present in the source.

The same pattern applies to out_file, which also has Windows-specific code but excluded tests.

Action needed: Manually verify whether Windows support is production-ready and either extend tests to cover Windows or disable the plugin on Windows if support is incomplete.

@patrick-stephens
Copy link
Collaborator

Once the coderabbit comments are addressed we should trigger a full package build but @SagiROosto can you confirm it builds for all targets?

@SagiROosto
Copy link
Author

@patrick-stephens UT failed for typo issue I Fixed it along with coderabbit suggestions.
This should work on all targets that out_file supports. Tho I'm not sure regarding windows because I use ZLIB for gzip compression

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
tests/runtime/out_logrotate.c (1)

294-354: Address the past review concern and clarify expected file count.

The addition of TEST_ASSERT(file_count >= 0) at lines 340 and 349 correctly addresses the past review concern about directory listing failures being silently ignored. Good fix!

However, there's an inconsistency in the comment at line 350: it states "Current file + 2 rotated files" but checks file_count <= 4. With max_files=3 configured at line 324, the expected count should be 1 current file + 3 rotated files = 4 total, not "2 rotated files" as the comment suggests.

Consider updating the comment to match the actual logic:

-    TEST_CHECK(file_count <= 4);  /* Current file + 2 rotated files */
+    TEST_CHECK(file_count <= 4);  /* Current file + 3 rotated files (max_files=3) */
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c7b361a and beffd1c.

📒 Files selected for processing (3)
  • plugins/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/logrotate.c (1 hunks)
  • tests/runtime/out_logrotate.c (1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.

Applied to files:

  • plugins/out_logrotate/logrotate.c
🔇 Additional comments (11)
plugins/CMakeLists.txt (1)

366-366: LGTM!

The plugin registration follows the established pattern and is correctly positioned alphabetically between out_file and out_forward.

tests/runtime/out_logrotate.c (3)

36-154: LGTM!

The helper functions are well-structured with appropriate error handling and resource cleanup. The recursive directory deletion, file counting, and pattern matching logic are all correct.


156-229: LGTM!

The basic rotation test correctly configures the plugin, triggers rotation by writing sufficient data, and validates that both the original file and rotated files exist.


231-292: LGTM!

The gzip compression test properly validates that rotated files are compressed by waiting for .gz files to appear, accounting for potential async compression.

plugins/out_logrotate/logrotate.c (7)

106-206: LGTM!

The initialization function correctly sets up the plugin context, creates the hash table for per-file size tracking, parses configuration options, and includes proper error handling with resource cleanup.


531-576: LGTM!

The file size tracking correctly uses a hash table to maintain per-file size state, addressing the previous concern about a shared global counter. The should_rotate_file function retrieves size by filename, and update_file_size_counter uses fstat to read actual file size before updating the hash table.


625-745: LGTM!

The streaming gzip compression implementation is memory-efficient and correct:

  • Processes files in 64KB chunks to avoid loading entire files into memory
  • Properly initializes deflate with raw deflate mode (negative window bits)
  • Writes RFC 1952 compliant gzip header and footer with CRC32
  • Verifies all input is consumed and compression completes with Z_STREAM_END
  • Includes proper error handling and resource cleanup

860-944: LGTM!

The cleanup logic correctly:

  • Counts rotated files matching the timestamp pattern
  • Returns early if within the max_files limit
  • Sorts files by modification time (oldest first) using a simple bubble sort
  • Removes excess files to maintain the max_files limit

The approach ensures that rotated files are cleaned up while preserving the current active file.


1067-1081: LGTM!

The metrics handling path now correctly calls update_file_size_counter(ctx, out_file, fp) at line 1078 before closing the file, which addresses the previous review concern about stale file size counters preventing rotation for metrics-only streams.


946-1182: LGTM!

The flush callback implements the complete workflow correctly:

  • Checks rotation condition using per-file size from hash table
  • Performs rotation and cleanup when needed
  • Removes rotated file's entry from hash table to reset tracking
  • Initializes file size from disk for new files using fstat
  • Updates file size counter after all write operations (logs, metrics, msgpack)

The per-file size tracking ensures correct behavior when multiple files are managed by a single plugin instance.


1184-1274: LGTM!

The exit function properly cleans up resources by destroying the hash table and freeing the context. The configuration map is comprehensive with sensible defaults: 100MB max size, 7 max files retained, and gzip compression enabled by default.

@patrick-stephens
Copy link
Collaborator

You'll have to fix the commits that don't match the guidelines btw, i.e. you need a plugin/module prefix like out_logrotate

@SagiROosto SagiROosto force-pushed the out_logrotate branch 2 times, most recently from 0dd42f9 to 99917b8 Compare November 5, 2025 15:45
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (8)
plugins/out_logrotate/logrotate.c (8)

129-134: Consider a larger hash table size for multi-file scenarios.

The hash table is created with only 64 buckets. If the plugin writes to many different files (when file is not set and tags are used as filenames), this could lead to excessive hash collisions and degraded performance. Consider using a larger size (e.g., 256 or 512) or making it configurable for high-throughput, multi-file deployments.


605-623: Document the 4GB limitation for gzip compression.

The gzip footer stores the original file size in 32 bits (4 bytes), which limits support to files under 4GB. While this is a standard gzip limitation, files larger than 4GB will have their size truncated modulo 2^32, which could cause decompression tools to report incorrect sizes or fail validation. Consider logging a warning when compressing files approaching or exceeding this limit, or documenting this limitation in the plugin configuration.


783-795: Check unlink() return values for error logging.

Both unlink() calls (lines 789 and 793) don't check return values. If deletion fails (e.g., permission denied), the files remain on disk but the function returns success or failure based solely on compression status. While the gzip failure path returns -1, the success path (line 790) doesn't verify that cleanup succeeded. Consider logging warnings if unlink() fails.

         if (ret == 0) {
             /* Remove the uncompressed file */
-            unlink(rotated_filename);
+            if (unlink(rotated_filename) != 0) {
+                flb_plg_warn(ctx->ins, "failed to remove uncompressed file %s", rotated_filename);
+            }
             flb_plg_debug(ctx->ins, "rotated and compressed file: %s", gzip_filename);
         } else {
             /* Remove the failed gzip file */
-            unlink(gzip_filename);
+            if (unlink(gzip_filename) != 0) {
+                flb_plg_warn(ctx->ins, "failed to remove incomplete gzip file %s", gzip_filename);
+            }
             return -1;
         }

911-923: Bubble sort may be inefficient for many rotated files.

The bubble sort has O(n²) complexity. If max_files is high and many files accumulate (e.g., hundreds of rotated logs), this could introduce noticeable latency during cleanup. For production environments with large retention counts, consider using qsort() with a comparison function for O(n log n) performance.


216-242: No validation for non-map objects in CSV output.

The function checks obj->type == MSGPACK_OBJECT_MAP && obj->via.map.size > 0 (line 216) but doesn't handle the case where the object is not a map or is empty. If a non-map object is passed, the function returns 0 (success) without writing anything. While this may be intentional, it could lead to silent data loss. Consider logging a warning when non-map objects are encountered.

 static int csv_output(FILE *fp, int column_names,
                       struct flb_time *tm, msgpack_object *obj,
                       struct flb_logrotate_conf *ctx)
 {
     int i;
     int map_size;
     msgpack_object_kv *kv = NULL;
 
-    if (obj->type == MSGPACK_OBJECT_MAP && obj->via.map.size > 0) {
+    if (obj->type != MSGPACK_OBJECT_MAP) {
+        flb_plg_warn(ctx->ins, "csv_output: expected map, got type %d", obj->type);
+        return -1;
+    }
+    
+    if (obj->via.map.size > 0) {
         kv = obj->via.map.ptr;
         map_size = obj->via.map.size;
         ...
+    } else {
+        flb_plg_debug(ctx->ins, "csv_output: empty map, skipping");
     }
     return 0;
 }

252-272: No validation for non-map objects in LTSV output.

Similar to csv_output, this function silently returns success if the object is not a map or is empty (line 252). Consider adding validation and logging for consistency with error handling elsewhere in the plugin.


370-381: Silent failure in plain_output when JSON conversion fails.

If flb_msgpack_to_json_str() returns NULL, the function silently skips writing and returns 0 (success). This could lead to data loss without any indication to the caller. Consider returning -1 or logging a warning when conversion fails.

 static int plain_output(FILE *fp, msgpack_object *obj, size_t alloc_size, int escape_unicode)
 {
     char *buf;
 
     buf = flb_msgpack_to_json_str(alloc_size, obj, escape_unicode);
     if (buf) {
         fprintf(fp, "%s" NEWLINE,
                 buf);
         flb_free(buf);
+    } else {
+        return -1;
     }
     return 0;
 }

383-407: print_metrics_text doesn't return error status.

The function is void and doesn't return error status to the caller. If cmt_decode_msgpack_create() fails (line 394), an error is logged but the caller (line 1075) has no way to know the write failed. Consider changing the return type to int and propagating the error so the flush operation can retry or handle the failure appropriately.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between beffd1c and 99917b8.

📒 Files selected for processing (3)
  • plugins/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/logrotate.c (1 hunks)
  • tests/runtime/out_logrotate.c (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/runtime/out_logrotate.c
  • plugins/CMakeLists.txt
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.

Applied to files:

  • plugins/out_logrotate/logrotate.c
🧬 Code graph analysis (1)
plugins/out_logrotate/logrotate.c (7)
include/fluent-bit/flb_mem.h (2)
  • flb_calloc (84-96)
  • flb_free (126-128)
src/flb_hash_table.c (5)
  • flb_hash_table_create (99-137)
  • flb_hash_table_destroy (197-215)
  • flb_hash_table_get (496-522)
  • flb_hash_table_add (401-494)
  • flb_hash_table_del (605-674)
include/fluent-bit/flb_output.h (1)
  • flb_output_config_map_set (1298-1321)
src/flb_output.c (2)
  • flb_output_get_property (1096-1099)
  • flb_output_set_context (1524-1527)
src/flb_pack.c (1)
  • flb_msgpack_to_json_str (1459-1500)
lib/cmetrics/src/cmt_decode_msgpack.c (1)
  • cmt_decode_msgpack_create (1450-1496)
src/flb_log_event_decoder.c (3)
  • flb_log_event_decoder_init (103-120)
  • flb_log_event_decoder_next (309-456)
  • flb_log_event_decoder_destroy (151-178)
🔇 Additional comments (1)
plugins/out_logrotate/logrotate.c (1)

1073-1081: Metrics file size update now included - excellent!

The past review noted that update_file_size_counter() was missing after writing metrics, causing rotation to never trigger for metrics-only streams. This has been correctly addressed at line 1078.

@SagiROosto SagiROosto force-pushed the out_logrotate branch 2 times, most recently from 2a77707 to 329cba4 Compare November 5, 2025 15:53
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 99917b8 and 329cba4.

📒 Files selected for processing (10)
  • CMakeLists.txt (1 hunks)
  • cmake/plugins_options.cmake (1 hunks)
  • cmake/windows-setup.cmake (1 hunks)
  • conf/fluent-bit-logrotate.conf (1 hunks)
  • plugins/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/logrotate.c (1 hunks)
  • plugins/out_logrotate/logrotate.h (1 hunks)
  • tests/runtime/CMakeLists.txt (1 hunks)
  • tests/runtime/out_logrotate.c (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (5)
  • cmake/plugins_options.cmake
  • tests/runtime/out_logrotate.c
  • conf/fluent-bit-logrotate.conf
  • plugins/out_logrotate/CMakeLists.txt
  • plugins/CMakeLists.txt
🧰 Additional context used
🧠 Learnings (3)
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.

Applied to files:

  • cmake/windows-setup.cmake
  • CMakeLists.txt
  • tests/runtime/CMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.

Applied to files:

  • cmake/windows-setup.cmake
  • CMakeLists.txt
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.

Applied to files:

  • plugins/out_logrotate/logrotate.c

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
plugins/out_logrotate/logrotate.c (2)

578-582: Optional: Remove redundant hash table delete.

Lines 579–581 delete the existing key before re-adding it, but flb_hash_table_add() already handles replacement internally (see src/flb_hash_table.c lines 430–442). The delete-then-add pattern adds unnecessary overhead—though it's safe, it doubles the hash lookups.

Apply this diff to streamline the update:

-        /* Delete existing entry if it exists to ensure we can update the value */
-        flb_hash_table_del(ctx->file_sizes, filename);
         /* Store or update file size in hash table */
         ret = flb_hash_table_add(ctx->file_sizes, filename, strlen(filename),
                                   &file_size, sizeof(size_t));

The flb_hash_table_add function already performs an in-place update for existing keys, so the explicit delete is not required.


783-795: Optional: Add path-length validation before constructing rotated filenames.

Lines 784 and 795 use snprintf with PATH_MAX - 1, which will silently truncate if filename is too long. While PATH_MAX is typically large (4096 on Linux), a truncated filename would cause rotated files to be unreachable by is_valid_rotation_filename() and cleanup_old_files(), breaking rotation cleanup.

Add a length check before constructing the rotated filename:

+    /* Validate that we have room for timestamp suffix */
+    if (strlen(filename) + 16 >= PATH_MAX) {  /* ".YYYYMMDD_HHMMSS" = 16 chars */
+        flb_plg_error(ctx->ins, "filename too long for rotation: %s", filename);
+        return -1;
+    }
+
     /* Create rotated filename with timestamp */
     snprintf(rotated_filename, PATH_MAX - 1, "%s.%s", filename, timestamp);
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 329cba4 and 9d1e142.

📒 Files selected for processing (4)
  • plugins/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/logrotate.c (1 hunks)
  • tests/runtime/out_logrotate.c (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • plugins/out_logrotate/CMakeLists.txt
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.

Applied to files:

  • plugins/out_logrotate/logrotate.c
🧬 Code graph analysis (2)
plugins/out_logrotate/logrotate.c (7)
include/fluent-bit/flb_mem.h (2)
  • flb_calloc (84-96)
  • flb_free (126-128)
src/flb_hash_table.c (5)
  • flb_hash_table_create (99-137)
  • flb_hash_table_destroy (197-215)
  • flb_hash_table_get (496-522)
  • flb_hash_table_del (605-674)
  • flb_hash_table_add (401-494)
include/fluent-bit/flb_output.h (1)
  • flb_output_config_map_set (1298-1321)
src/flb_output.c (2)
  • flb_output_get_property (1096-1099)
  • flb_output_set_context (1524-1527)
src/flb_pack.c (1)
  • flb_msgpack_to_json_str (1459-1500)
lib/cmetrics/src/cmt_decode_msgpack.c (1)
  • cmt_decode_msgpack_create (1450-1496)
src/flb_log_event_decoder.c (3)
  • flb_log_event_decoder_init (103-120)
  • flb_log_event_decoder_next (309-456)
  • flb_log_event_decoder_destroy (151-178)
tests/runtime/out_logrotate.c (1)
src/flb_lib.c (10)
  • flb_create (143-225)
  • flb_service_set (652-678)
  • flb_input (266-276)
  • flb_input_set (305-335)
  • flb_output (279-289)
  • flb_output_set (520-551)
  • flb_start (983-994)
  • flb_lib_push (843-870)
  • flb_stop (1011-1055)
  • flb_destroy (228-263)
🔇 Additional comments (4)
plugins/CMakeLists.txt (1)

366-366: LGTM: Plugin registration is correct.

The out_logrotate plugin is registered following the same pattern as other output plugins, and the placement maintains alphabetical order.

tests/runtime/out_logrotate.c (3)

157-235: LGTM: Basic rotation test is well-structured.

The test properly:

  • Sets up and tears down the test directory
  • Configures rotation with max_size=5K and gzip=false
  • Writes enough data (~16KB) to trigger rotation
  • Verifies both the current file and at least one rotated file exist

The sleep intervals (lines 210, 218) are necessary to allow async flush operations to complete.


237-300: LGTM: Gzip compression test validates the compression feature.

The test correctly:

  • Enables gzip compression with gzip=true
  • Triggers rotation by writing sufficient data
  • Uses wait_for_file_pattern() to poll for rotated files with .gz suffix
  • Verifies that gzip compression was applied to rotated files

302-366: LGTM: Max files cleanup test validates the retention policy.

The test properly:

  • Configures max_files=3 to limit retained rotated files
  • Triggers multiple rotations by writing data in 5 iterations
  • Validates that file count never exceeds 4 (current file + 3 rotated files)
  • Uses TEST_ASSERT(file_count >= 0) to catch directory listing failures (addressing the past review comment)
  • Verifies the final count after all rotations complete

The iterative checking (lines 349–351) ensures cleanup happens progressively during rotation, not just at the end.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
plugins/out_logrotate/logrotate.c (2)

607-654: Consider logging when file size update fails.

The function silently returns if fstat() fails or reports a negative size (line 615-617). While this prevents crashes, it makes troubleshooting harder when file size tracking silently fails to update. Consider adding a debug or warning message to aid diagnostics:

   if (fstat(fileno(fp), &st) != 0 || st.st_size < 0) {
+    flb_plg_debug(ctx->ins, "failed to stat file %s for size update: %s",
+                  filename, fstat(fileno(fp), &st) != 0 ? strerror(errno) : "negative size");
     return;
   }

112-209: Consider adding max_size validation to prevent confusing behavior.

While max_files is validated (lines 141-146), there's no validation for max_size. If a user accidentally sets max_size=0, the rotation check at line 1248 (file_size >= ctx->max_size) will always be true, triggering rotation on every flush before any data is written. This results in repeated failed rename() calls (file doesn't exist yet) and many error messages.

Consider adding validation in cb_logrotate_init() similar to max_files:

   if (ctx->max_files <= 0) {
     flb_plg_error(ctx->ins, "invalid max_files=%d; must be >= 1",
                   ctx->max_files);
     flb_free(ctx);
     return -1;
   }

+  if (ctx->max_size == 0) {
+    flb_plg_error(ctx->ins, "invalid max_size=0; must be > 0");
+    flb_free(ctx);
+    return -1;
+  }
+
   /* Optional, file format */
   tmp = flb_output_get_property("Format", ins);

Alternatively, treat max_size == 0 as "no rotation" by skipping the rotation block entirely when max_size is 0.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between cf9992a and e0ca66a.

📒 Files selected for processing (2)
  • plugins/out_logrotate/logrotate.c (1 hunks)
  • tests/runtime/out_logrotate.c (1 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.

Applied to files:

  • plugins/out_logrotate/logrotate.c
🧬 Code graph analysis (2)
plugins/out_logrotate/logrotate.c (6)
include/fluent-bit/flb_mem.h (2)
  • flb_calloc (84-96)
  • flb_free (126-128)
src/flb_output.c (2)
  • flb_output_get_property (1108-1111)
  • flb_output_set_context (1546-1549)
src/flb_pack.c (1)
  • flb_msgpack_to_json_str (1459-1500)
lib/cmetrics/src/cmt_decode_msgpack.c (1)
  • cmt_decode_msgpack_create (1450-1496)
src/flb_lock.c (4)
  • flb_lock_init (6-17)
  • flb_lock_destroy (19-30)
  • flb_lock_acquire (32-65)
  • flb_lock_release (67-100)
src/flb_log_event_decoder.c (2)
  • flb_log_event_decoder_init (103-120)
  • flb_log_event_decoder_next (309-456)
tests/runtime/out_logrotate.c (2)
include/fluent-bit/flb_mem.h (1)
  • flb_free (126-128)
src/flb_lib.c (2)
  • flb_start (983-994)
  • flb_stop (1011-1055)
🪛 ast-grep (0.40.0)
tests/runtime/out_logrotate.c

[warning] 452-452: A check is done with stat and then the file is used. There is no guarantee that the status of the file has not changed since the call to stat which may allow attackers to bypass permission checks.
Context: fopen
Note: [CWE-367]: Time-of-check Time-of-use (TOCTOU) Race Condition [REFERENCES]
- https://wiki.sei.cmu.edu/confluence/display/c/FIO45-C.+Avoid+TOCTOU+race+conditions+while+accessing+files

(file-stat-before-action-c)

🔇 Additional comments (10)
tests/runtime/out_logrotate.c (6)

65-131: LGTM: Robust recursive directory deletion.

The helper properly validates inputs, handles non-existent paths gracefully, and recursively processes subdirectories. Error handling is appropriate for a test utility.


133-152: LGTM: Clean file counting implementation.

The helper correctly counts files matching a prefix and properly signals errors with -1 return value.


154-185: LGTM: Effective polling-based file wait.

The polling approach with 1-second intervals and timeout is appropriate for test scenarios. The pattern matching logic correctly checks both prefix and suffix.


187-220: LGTM: Proper file reading with good error handling.

The function correctly allocates buffers, validates reads, and cleans up resources on all error paths.


449-466: TOCTOU in test context is acceptable.

Static analysis flagged a time-of-check-time-of-use pattern where stat() is called before fopen(). In this test scenario, the TOCTOU is not a concern:

  • Tests execute in isolated environments
  • The check verifies file existence and size before reading
  • No security boundary is crossed

This pattern is appropriate for test validation.


891-1007: LGTM: Well-structured multithreaded test with proper synchronization.

The test correctly:

  • Uses pthread mutex to protect shared success flag
  • Waits for all threads to complete before verification
  • Allows sufficient time for flush cycles (3 seconds)
  • Verifies data integrity by counting records in output

The synchronization via mutex at lines 879-881 prevents race conditions when updating the shared success flag.

plugins/out_logrotate/logrotate.c (4)

807-837: LGTM: Z_FINISH issue correctly fixed.

The additional finalization block at lines 812-837 properly addresses the past review concern about files with sizes that are exact multiples of GZIP_CHUNK_SIZE. The logic ensures that:

  1. If the main loop completes compression (status == Z_STREAM_END), no additional work is needed
  2. If the main loop exits due to bytes_read == 0 before sending Z_FINISH, this block sends it
  3. All compressed output is written before footer generation

This guarantees valid gzip streams for all file sizes.


1174-1524: LGTM: Mutex handling is correctly implemented throughout flush.

The function uses a complex but correct lock acquisition and release pattern:

  • Lock is acquired at line 1230 when entry exists
  • Lock is released before rotation (line 1277-1281) and re-acquired after (line 1304-1310)
  • Every return path either has lock_acquired == 0 or properly releases the lock before returning
  • The lock_acquired flag correctly tracks lock state across the function

All potential deadlocks and resource leaks are avoided.


875-952: LGTM: Rotation logic with proper locking and error handling.

The rotate_file() function correctly:

  • Acquires lock before checking size (lines 890-895)
  • Double-checks rotation is still needed to handle races (lines 898-902)
  • Generates unique timestamp-based filenames (line 910-913)
  • Optionally compresses rotated files (lines 924-938)
  • Releases lock on all paths (lines 945-949)

The race condition check at line 898 prevents unnecessary rotations if another thread already rotated the file.


1011-1172: LGTM: File cleanup with robust sorting and validation.

The cleanup function properly:

  • Validates rotation filename format using strict pattern matching (lines 960-1009)
  • Handles both Windows and POSIX directory traversal (lines 1025-1134)
  • Sorts files by modification time to remove oldest first (lines 1137-1149)
  • Respects the max_files limit (lines 1152-1163)

The pattern validation prevents accidental deletion of non-rotated files.

SagiROosto and others added 19 commits November 26, 2025 11:40
* add logrotate out plugin

* fix gzip

* fix test

* update max_size

* update gzip

* add logs

Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
…d fix UT

Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
…ad of hash table

This change replaces the hash table with a linked list for storing file sizes, improving memory management and simplifying the code. Additionally, helper functions for managing file size entries have been added, enhancing the overall structure and readability of the logrotate plugin.

Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
…ng support

This commit introduces several new test functions for various log formats including CSV, LTSV, plain text, MsgPack, and template formats. Additionally, it enhances the multithreading capabilities of the logrotate tests, ensuring that data is correctly handled and written in a concurrent environment. The changes also include helper functions for reading file content and verifying the output format.

Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
…and improve directory creation and rotation logic.

Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
… MSVC builds.

Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
…ssion

Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
Signed-off-by: SagiROosto <sagi.rosenthal@oosto.com>
@SagiROosto
Copy link
Author

crossing fingers 🤞🤞🤞

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (1)
plugins/out_logrotate/logrotate.c (1)

141-152: Validate max_size on init to avoid pathological “rotate every flush” configs

max_size is read via FLB_CONFIG_MAP_SIZE and logged, but unlike max_files it is not validated. A user typo like max_size 0 would make:

if (file_size >= ctx->max_size)

always true, so every flush attempts rotation (often failing rename() initially, then rotating on every write once the file exists). It’s not memory‑unsafe, but it’s very surprising behavior for a simple misconfiguration.

You already reject invalid max_files; I’d mirror that for max_size:

   ret = flb_output_config_map_set(ins, (void *)ctx);
   if (ret == -1) {
     flb_free(ctx);
     return -1;
   }

+  /* Validate rotation limits */
+  if (ctx->max_size == 0) {
+    flb_plg_error(ctx->ins,
+                  "invalid max_size=%zu; must be > 0", ctx->max_size);
+    flb_free(ctx);
+    return -1;
+  }
+
   if (ctx->max_files <= 0) {
     flb_plg_error(ctx->ins, "invalid max_files=%d; must be >= 1",
                   ctx->max_files);
     flb_free(ctx);
     return -1;
   }

This keeps behavior well‑defined and surfaces misconfigurations early.

If you prefer, you could instead treat max_size == 0 as “rotation disabled” and document that explicitly; in either case, clamping/validation should be consistent with how other outputs treat size limits.

Also applies to: 199-207, 1734-1736

🧹 Nitpick comments (3)
tests/runtime/out_logrotate.c (3)

196-214: Consider adding a note about POSIX-only dependency.

The count_files_in_directory helper uses POSIX opendir/readdir without a Windows implementation, unlike recursive_delete_directory which has both. This is acceptable since the test is only compiled on non-Windows systems, but adding a brief comment noting this would improve clarity for future maintainers.


1365-1366: Consider using a portable format for time_t.

The %lu format specifier assumes time_t is unsigned long, but this is not guaranteed across all platforms. For better portability, cast explicitly:

-  snprintf(json_payload, json_size, "[%lu, {\"message\": \"%s\"}]", time(NULL),
+  snprintf(json_payload, json_size, "[%ld, {\"message\": \"%s\"}]", (long)time(NULL),
           large_message);

This is a minor issue for test code but would prevent potential warnings on some platforms.


961-965: Minor: Array size exceeds usage.

The threads and thread_data arrays are sized for 8 elements but only 4 are used (num_threads = 4). Consider matching the array size to num_threads or using a #define for consistency:

+#define NUM_THREADS 4
+#define EVENTS_PER_THREAD 10
 ...
-  pthread_t threads[8];
-  struct thread_data thread_data[8];
+  pthread_t threads[NUM_THREADS];
+  struct thread_data thread_data[NUM_THREADS];
   pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
   int success = 1;
-  int num_threads = 4;
-  int events_per_thread = 10;
+  int num_threads = NUM_THREADS;
+  int events_per_thread = EVENTS_PER_THREAD;
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e0ca66a and e7a23ae.

📒 Files selected for processing (10)
  • CMakeLists.txt (1 hunks)
  • cmake/plugins_options.cmake (1 hunks)
  • cmake/windows-setup.cmake (1 hunks)
  • conf/fluent-bit-logrotate.conf (1 hunks)
  • plugins/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/CMakeLists.txt (1 hunks)
  • plugins/out_logrotate/logrotate.c (1 hunks)
  • plugins/out_logrotate/logrotate.h (1 hunks)
  • tests/runtime/CMakeLists.txt (1 hunks)
  • tests/runtime/out_logrotate.c (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (4)
  • plugins/out_logrotate/CMakeLists.txt
  • cmake/plugins_options.cmake
  • conf/fluent-bit-logrotate.conf
  • plugins/out_logrotate/logrotate.h
🧰 Additional context used
🧠 Learnings (3)
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.

Applied to files:

  • tests/runtime/CMakeLists.txt
  • cmake/windows-setup.cmake
  • CMakeLists.txt
📚 Learning: 2025-08-31T12:46:11.940Z
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.

Applied to files:

  • CMakeLists.txt
📚 Learning: 2025-10-23T07:43:16.216Z
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11059
File: plugins/in_tail/tail_file.c:1618-1640
Timestamp: 2025-10-23T07:43:16.216Z
Learning: In plugins/in_tail/tail_file.c, when truncate_long_lines is enabled and the buffer is full, the early truncation path uses `lines > 0` as the validation pattern to confirm whether process_content successfully processed content. This is intentional to track occurrences of line processing rather than byte consumption, and consuming bytes based on `processed_bytes > 0` would be overkill for this validation purpose.

Applied to files:

  • plugins/out_logrotate/logrotate.c
🧬 Code graph analysis (2)
tests/runtime/out_logrotate.c (2)
include/fluent-bit/flb_mem.h (1)
  • flb_free (126-128)
src/flb_lib.c (10)
  • flb_create (143-225)
  • flb_service_set (652-678)
  • flb_input (266-276)
  • flb_input_set (305-335)
  • flb_output (279-289)
  • flb_output_set (520-551)
  • flb_start (983-994)
  • flb_lib_push (843-870)
  • flb_stop (1011-1055)
  • flb_destroy (228-263)
plugins/out_logrotate/logrotate.c (3)
src/flb_lock.c (4)
  • flb_lock_init (6-17)
  • flb_lock_acquire (32-65)
  • flb_lock_release (67-100)
  • flb_lock_destroy (19-30)
src/flb_output.c (2)
  • flb_output_get_property (1108-1111)
  • flb_output_set_context (1546-1549)
src/flb_log_event_decoder.c (2)
  • flb_log_event_decoder_init (103-120)
  • flb_log_event_decoder_next (309-456)
🪛 ast-grep (0.40.0)
tests/runtime/out_logrotate.c

[warning] 514-514: A check is done with stat and then the file is used. There is no guarantee that the status of the file has not changed since the call to stat which may allow attackers to bypass permission checks.
Context: fopen
Note: [CWE-367]: Time-of-check Time-of-use (TOCTOU) Race Condition [REFERENCES]
- https://wiki.sei.cmu.edu/confluence/display/c/FIO45-C.+Avoid+TOCTOU+race+conditions+while+accessing+files

(file-stat-before-action-c)

🔇 Additional comments (11)
plugins/CMakeLists.txt (1)

366-366: LGTM!

The plugin registration follows the established pattern and is appropriately placed after out_file in the OUTPUTS section.

tests/runtime/CMakeLists.txt (1)

237-237: LGTM!

The test registration correctly follows the pattern used for out_file.c and is appropriately placed within the non-Windows block, since the test file uses POSIX-specific functions like opendir/readdir.

CMakeLists.txt (1)

302-302: LGTM!

The FLB_OUT_LOGROTATE flag is correctly added under the FLB_ALL block, following the established pattern for output plugins.

cmake/windows-setup.cmake (1)

103-103: Verify zlib availability for gzip compression on Windows.

The logrotate plugin is enabled on Windows, but the PR comments indicate uncertainty about Windows support because gzip relies on zlib. Please confirm that zlib is available in Windows builds and that the gzip compression functionality works correctly on Windows targets.

tests/runtime/out_logrotate.c (5)

1-25: LGTM!

The includes and test data imports are well-organized with appropriate conditional compilation for Windows vs POSIX systems.


511-527: Static analysis TOCTOU warning is acceptable for test code.

The static analysis flagged a TOCTOU race between stat() and fopen(). In a controlled test environment, this is not a security concern. The pattern is appropriate for validating msgpack binary output.


1254-1268: Past review feedback properly addressed.

The guard against count_files_in_directory returning -1 has been correctly implemented with TEST_ASSERT(file_count >= 0) before the upper-bound check, ensuring directory listing failures will now properly fail the test.


44-63: Comprehensive test coverage.

The test suite provides thorough coverage of the logrotate plugin's functionality including:

  • Basic rotation and gzip compression
  • Max files cleanup and validation
  • Multiple output formats (CSV, LTSV, plain, msgpack, template)
  • Configuration options (path, mkdir, delimiters)
  • Multithreaded scenarios

The test structure is clean with proper setup/teardown patterns.


1022-1023: Unable to verify wait_for_file function availability due to repository access issues.

The repository clone failed during verification attempts, preventing inspection of the test framework headers and source files. Manual verification is needed to confirm that wait_for_file is properly defined and exported by the test infrastructure (likely in flb_tests_runtime.h or a linked test utility).

plugins/out_logrotate/logrotate.c (2)

1308-1345: Unfortunately, I'm unable to access the repository to verify the code details. The execution environment cannot clone the fluent/fluent-bit repository or access it via the GitHub API.

Given this constraint, I can only work with the information provided in the review comment itself. The analysis presented is logically sound in its reasoning about mutex lock/unlock patterns:

  1. Unbalanced unlock issue: The logic that you release a lock before calling a function, then unconditionally release it again without tracking state, would indeed cause undefined behavior if the second acquire fails.

  2. Nested lock acquisition: Calling update_file_size() (which acquires ctx->list_lock) while already holding that lock would cause a deadlock or failure.

  3. Proposed fix: Tracking list_lock_acquired state to conditionally release is a standard and correct approach.

However, without direct code verification, I cannot:

  • Confirm the exact current code state matches the provided snippet
  • Verify the post-rotation block (lines 1411-1436) exhibits the claimed issues
  • Ensure the diff is complete and there are no other affected paths
  • Validate the fix logic handles all edge cases


1527-1546: Based on my verification of standard C library semantics for fwrite(), the review comment's technical analysis is accurate:

Verified facts:

  1. fwrite() returns size_t (number of items written), never negative—the ret < 0 check will never trigger, masking write errors
  2. ✅ When fwrite() returns 0 with ferror(fp) set, total never increments, creating an infinite loop
  3. ✅ The code does not advance the buffer pointer (off remains 0), so short writes cause the same data to be re-written, potentially duplicating data
  4. ✅ Proper error detection requires checking ferror(fp) after fwrite() returns, not checking for negative values

The standard C library documentation confirms that:

  • Partial writes are possible and indicate an error condition or unspecified stream state
  • Attempting to retry fwrite() on a short write without proper error handling can cause data loss or duplication due to stdio buffering
  • The recommended pattern is to check return value against expected items written and use ferror() for error detection

Since I cannot access the repository to examine the actual code context, I cannot fully verify all implementation details. However, the core technical claims in the review are supported by standard C semantics and represent genuine safety issues.


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-required ok-package-test Run PR packaging tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants