Resolved: Incorrect connection count handling pianod crash logs showing connection from 0.0.0.0 In the last seconds before the 2021-01-30 crash 2021-01-30 07:22:20: fb_service.c:337 (fb_validate_connection): #16: Connection from 127.0.0.1 permitted. 2021-01-30 07:22:20: fb_event.c:530 (fb_new_connect): #16: New cleartext IP4 line connection from 127.0.0.1 port 60330 2021-01-30 07:22:20: fb_service.c:536 (fb_destroy_connection): #16: Connection terminated. 2021-01-30 07:22:20: engine.cpp:442 PurgeUnselectedSongs (LOG_GENERAL): Removing song 3ss390771106-11 ('Erstes Dreigroschen-Finale') from random queue: song's playlist is not in the mix 2021-01-30 07:22:20: engine.cpp:442 PurgeUnselectedSongs (LOG_GENERAL): Removing song 3ss537272334-1 ('Tchaikovsky - Concerto in D, Op. 35 - Allegro moderato - Heifetz') from random queue: song's playlist is not in the mix 2021-01-30 07:22:20: engine.cpp:442 PurgeUnselectedSongs Crash #2 on 2021-02-02, last seconds: 2021-02-02 05:49:32: fb_http.c:1108 (http_serve_data): #20: pianod: HTTP request: GET translate.js (200 Ok) 2021-02-02 05:49:32: fb_http.c:1108 (http_serve_data): #21: pianod: HTTP request: GET client.lib.js (200 Ok) 2021-02-02 05:49:32: fb_http.c:1108 (http_serve_data): #20: pianod: HTTP request: GET no-art.jpeg (200 Ok) 2021-02-02 05:49:32: fb_service.c:337 (fb_validate_connection): #22: Connection from 10.69.69.254 permitted. 2021-02-02 05:49:32: fb_event.c:529 (fb_new_connect): #22: New cleartext IP4 HTTP connection from 10.69.69.254 port 37852 2021-02-02 05:49:32: fb_http.c:775 (fb_greet_websocket): #22: pianod WebSocket session initiated 2021-02-02 05:49:32: fb_http.c:1108 (http_serve_data): #21: pianod: HTTP request: GET pianod-button.gif (200 Ok) 2021-02-02 05:49:32: fb_http.c:1108 (http_serve_data): #20: pianod: HTTP request: GET pianod-icon.gif (200 Ok) 2021-02-02 05:49:32: fb_service.c:536 (fb_destroy_connection): #20: Connection terminated. Similarities: - Both crashes are preceeded by fb_destroy_connection - Both crashes are in the same function, same line Differences: - The 2021-01-30 crash, fb_destroy_connection is closure of a line connection to pianod. - The 2021-01-30 crash occurs from periodic(), not from the command dispatcher. - The 2021-02-02 crash is closing an HTTP connection that is connected to Football only. - The 2021-02-02 crash occurs during a command dispatch. Crash happening: bool User::online (const PianodService &service) const { std::vector connections; for (auto conn : service) { if (conn->user == this) { ///< CRASH HAPPENING HERE return true; } } return false; } This is a pointer comparison, suggesting `conn` is either NULL or invalid. Could the iterator have returned something invalid? - Iterator only yields members with a non-null related object. - Related object is destroyed and properly nulled out in fb_arena.cpp case handler. - Related object is parts of FB_CONNECTION structure, which is `calloc`ed and thus should be NULL to start. - Football's C++ connection object gets destroyed. - So what removes the connection from the service? - The iterator is looking at the C FB_SERVICE structure. - relatedObject is set in getFromOld, built by service's allocNewConnection() in Football::Service. - Connection is removed during fb_destroy_connection on connection, which is subsequent to the FB_EVENT_CLOSE. This should not pose a problem. Rx -- Found issue in fb_accept_connection: service->connections [service->connection_count++] = connection; Insertion of service into socket preceeded `if (fb_validate_connection) ...`. If fb_validate_connection returned false, socket was closed and clean up done, but service's connection count not decremented, leaving a rogue pointer in the service's connections. Moved insertion to within `if (fb_validate_connection) ...` Fix release in pianod2 r362 Version info: pianod2 version 360 Build options: Switches: --enable-debug --with-compression SSL package: gnutls Sources: pandora curl compile version 7.74.0 curl runtime version 7.74.0 tonegenerator filesystem Media libraries: ffmpeg/libav (seems to be ffmpeg) avformat compile version 58.29.100 runtime version 58.45.100 avcodec compile version 58.54.100 runtime version 58.91.100 avfilter compile version 7.57.100 runtime version 7.85.100 avutil compile version 56.31.100 runtime version 56.51.100 taglib compile version 1.9.1 Audio outputs: libao libavdevice compile version 58.8.100 runtime version 58.10.100