I created msoak as a utility to subscribe to multiple MQTT brokers simultaneously and consume any number of topics; it soaks up what they have to offer and prints the topic/payload to stdout. I pipe that output to tinylog for storage, compression, and file rotation. I’ve been using this setup to log from a bunch of brokers ever since msoak was born two years ago.
Today, while looking at previously-logged data in preparation for consumption of future similar data, I discovered to my horror, that tinylog(8) truncates lines at what I first assumed was 1024 but actually is 1000 bytes.
I simply hadn’t noticed because most of the payloads we save are less than 1000 bytes. It’s my fault, as it’s documented, but I’d overlooked that sentence.
tinylog ignores empty lines, truncates lines longer than 1000 characters
Within minutes I found the definition and I’m now good. (80k is really sufficient for the moment.)
--- perp-2.07/perp/tinylog.c 2013-01-11 16:10:32.000000000 +0100
+++ perp-2.07-jp/perp/tinylog.c 2022-05-25 17:53:35.000000000 +0200
@@ -96,7 +96,7 @@
/* default maximum size for log file (bytes): */
#define CURRENT_MAX 100000
/* maximum size for line in log file (bytes): */
-#define LOGLINE_MAX 1000
+#define LOGLINE_MAX 80000
/* pause on exceptional error (billionths of second): */
#define EPAUSE 555444321UL
They say a backup is only as good as its recovery. Very true.