Reduce binary footprint: optional curl HTTP client and minimal SQLite#1499
Reduce binary footprint: optional curl HTTP client and minimal SQLite#1499bmehta001 wants to merge 4 commits into
Conversation
On the CPP11/curl path (non-Apple, non-Windows), the built-in libcurl HTTP client was always compiled and curl was a hard find_package(CURL REQUIRED) dependency -- pulling in curl and a TLS backend (OpenSSL/mbedTLS) even for hosts that already have their own HTTP stack. Add option(BUILD_CURL_HTTP_CLIENT ON). When OFF, the curl block is skipped (no find_package(CURL), no link, no -DHAVE_MAT_CURL_HTTP_CLIENT) and the build instead defines -DMATSDK_NO_DEFAULT_HTTP_CLIENT. mat/config.h then undefines HAVE_MAT_DEFAULT_HTTP_CLIENT centrally (regardless of the config preset), which the SDK already handles end-to-end: HttpClientFactory and HttpClient_Curl.cpp compile out, and LogManagerImpl's existing !HAVE_MAT_DEFAULT_HTTP_CLIENT branch requires the host to supply an IHttpClient via CFG_MODULE_HTTP_CLIENT. Default ON keeps existing behavior unchanged. Apple/Windows are unaffected (they use native HTTP stacks and never enter the curl block). Validated on WSL x64-linux: with OFF, libmat has no curl symbols and a consumer links with no -lcurl/-lTLS (1.43 MB stripped, vs 4.39 MB with curl+mbedTLS and 10.65 MB with curl+OpenSSL). Files changed: - CMakeLists.txt: BUILD_CURL_HTTP_CLIENT option + gating - lib/include/mat/config.h: central HAVE_MAT_DEFAULT_HTTP_CLIENT opt-out Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tprint The SDK uses SQLite only for its offline event-storage cache (plain tables, transactions, WAL, autovacuum/VACUUM, a few PRAGMAs, and one custom UTF-8 function), so most SQLite subsystems are dead weight. Add an option to compile a private SQLite from the vendored amalgamation with a set of amalgamation-safe strip flags (single source of truth: MATSDK_SQLITE_MINIMAL_DEFS), removing the external sqlite3 dependency and shrinking SQLite ~10.2% (.text) / ~12.5% (object). - Root CMakeLists.txt: add option(MATSDK_MINIMAL_SQLITE) (default OFF). In vcpkg mode, skip find_package(unofficial-sqlite3) when bundling, and emit a clear FATAL_ERROR pointing at the system-sqlite/minimal-sqlite features when neither provides SQLite (e.g. a bare [core] install). - lib/CMakeLists.txt: define MATSDK_SQLITE_MINIMAL_DEFS, compute MATSDK_BUNDLE_SQLITE (minimal OR vendored-Android), and build a single sqlite3_bundled. The strip flags are applied ONLY when MATSDK_MINIMAL_SQLITE is ON, so the default Android legacy build keeps its existing unstripped bundled SQLite. Warnings are disabled on the vendored target (/w on MSVC, -w on GCC/Clang for the stripped build) so the SDK's -Werror/-WX does not fire on amalgamation code. A static mat propagates the PRIVATE sqlite3_bundled through its link interface, so export+install it. - MSTelemetryConfig.cmake.in: skip find_dependency(unofficial-sqlite3) when bundled. - vcpkg port: add a minimal-sqlite feature (-DMATSDK_MINIMAL_SQLITE=ON) and move sqlite3 into a default system-sqlite feature so [core,minimal-sqlite] drops it. - docs/building-with-vcpkg.md: document the feature, the size win, and the static-absorption symbol-visibility caveat. SQLITE_OMIT_AUTOINIT and SQLITE_DEFAULT_MEMSTATUS=0 are deliberately NOT stripped: the former because skipSqliteInitAndShutdown lets the host skip the SDK's explicit sqlite3_initialize() (which the host cannot do against a private SQLite), the latter because the SDK arms a soft heap limit via sqlite3_soft_heap_limit64() that is only enforced while memory statistics are enabled. Validated: vendored Linux Debug (77 offline-storage/SQLite unit tests pass on the debug amalgamation), vcpkg [core,minimal-sqlite] consumer (links MSTelemetry::sqlite3_bundled, runs 10/10, external sqlite3 dropped), default vcpkg path regression (system-sqlite intact), and MSVC compile/link of sqlite3_bundled+mat. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces opt-in build knobs intended to reduce the SDK’s footprint in host-embedded scenarios by (1) allowing builds without the built-in libcurl HTTP client and (2) optionally replacing the external SQLite dependency with a private, feature-stripped bundled SQLite.
Changes:
- Added
BUILD_CURL_HTTP_CLIENTto omit the built-in curl/TLS HTTP client and require a host-suppliedIHttpClient. - Added
MATSDK_MINIMAL_SQLITEand corresponding vcpkgminimal-sqlitefeature to build a stripped, bundled SQLite instead of depending on vcpkg’ssqlite3. - Updated the vcpkg port and documentation to reflect the new SQLite feature split and minimal-SQLite option.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/ports/cpp-client-telemetry/vcpkg.json | Moves sqlite3 into a default system-sqlite feature and adds minimal-sqlite. |
| tools/ports/cpp-client-telemetry/portfile.cmake | Wires vcpkg minimal-sqlite feature into -DMATSDK_MINIMAL_SQLITE=ON. |
| lib/include/mat/config.h | Adds MATSDK_NO_DEFAULT_HTTP_CLIENT override to disable HAVE_MAT_DEFAULT_HTTP_CLIENT. |
| lib/CMakeLists.txt | Implements bundled/minimal SQLite build, link selection, and installation/export logic. |
| docs/building-with-vcpkg.md | Documents the new vcpkg minimal-sqlite feature and behavior. |
| CMakeLists.txt | Adds MATSDK_MINIMAL_SQLITE and BUILD_CURL_HTTP_CLIENT; adjusts vcpkg SQLite discovery flow. |
| cmake/MSTelemetryConfig.cmake.in | Skips unofficial-sqlite3 dependency discovery when SQLite is bundled. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if(MATSDK_BUNDLE_SQLITE) | ||
| # Private minimal SQLite instead of the vcpkg sqlite3 package. PRIVATE so the | ||
| # SDK-internal copy is not re-exported to consumers. | ||
| target_link_libraries(mat PRIVATE sqlite3_bundled) | ||
| else() | ||
| target_link_libraries(mat PUBLIC unofficial::sqlite3::sqlite3) | ||
| endif() |
There was a problem hiding this comment.
This is the static-library case CMake already handles: target_link_libraries(mat PRIVATE sqlite3_bundled) on a static mat places the dependency in INTERFACE_LINK_LIBRARIES wrapped in \$<LINK_ONLY:...>, so a downstream target that links only MSTelemetry::mat still pulls in MSTelemetry::sqlite3_bundled automatically (CMake propagates a static library's PRIVATE link deps because the archive itself does no linking). It is exported in the same set for exactly this reason.
Verified empirically: a vcpkg [core,minimal-sqlite] consumer that links only MSTelemetry::mat (x64-linux, static) builds, links, and runs 10/10 with no unresolved sqlite3_* symbols; the installed MSTelemetryTargets.cmake shows INTERFACE_LINK_LIBRARIES "\$<LINK_ONLY:MSTelemetry::sqlite3_bundled>;...". Making it PUBLIC would instead propagate the target's usage requirements (include dirs/defs) as a public contract, which is not wanted for an internal impl detail. Keeping it PRIVATE + LINK_ONLY is the correct semantic.
Comments 2 and 3 are addressed in 0977e64 (sqlite3_bundled is now exported only for a static mat; the status message reflects the SQLite path).
| # A static libmat propagates its PRIVATE static dependencies through its link | ||
| # interface, so the bundled SQLite must be part of the same export set and be | ||
| # installed alongside mat for downstream find_package() consumers to link. | ||
| set(MATSDK_INSTALL_TARGETS mat) | ||
| if(MATSDK_BUNDLE_SQLITE AND TARGET sqlite3_bundled) | ||
| list(APPEND MATSDK_INSTALL_TARGETS sqlite3_bundled) | ||
| endif() |
| find_package(ZLIB REQUIRED) | ||
| find_package(nlohmann_json CONFIG REQUIRED) | ||
| message(STATUS "Using vcpkg-provided sqlite3, zlib, nlohmann-json") | ||
| message(STATUS "Using vcpkg-provided zlib, nlohmann-json") |
- lib/CMakeLists.txt: only add sqlite3_bundled to the install/export set when mat is a STATIC_LIBRARY. A shared mat absorbs the private SQLite into libmat and does not propagate the PRIVATE dependency, so exporting the separate archive there was unnecessary and could let a consumer link a second SQLite copy. For a static mat the archive must stay exported because the static library propagates its PRIVATE dependency through its link interface (\$<LINK_ONLY:...>). - CMakeLists.txt: make the vcpkg dependency-mode status message reflect whether the external sqlite3 package or the private minimal SQLite is used. Verified with an isolated CMake export test: static mat exports m+sq (consumer linking only the namespaced lib resolves sq); shared mat exports only m and install(EXPORT) succeeds with sq excluded. Re-ran the vcpkg [core,minimal-sqlite] consumer (static x64-linux): 10/10. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| # Private minimal SQLite instead of the vcpkg sqlite3 package. PRIVATE so the | ||
| # SDK-internal copy is not re-exported to consumers. | ||
| target_link_libraries(mat PRIVATE sqlite3_bundled) |
| # minimal-sqlite feature -> build a private feature-stripped SQLite instead of | ||
| # linking the external sqlite3 package. Maps to -DMATSDK_MINIMAL_SQLITE=ON/OFF. | ||
| vcpkg_check_features( | ||
| OUT_FEATURE_OPTIONS FEATURE_OPTIONS | ||
| FEATURES | ||
| minimal-sqlite MATSDK_MINIMAL_SQLITE | ||
| ) |
There was a problem hiding this comment.
Intentionally not exposed as a vcpkg feature. minimal-sqlite is a transparent, drop-in change (it swaps one SQLite implementation for another with identical behavior), so it is safe to toggle from a manifest. BUILD_CURL_HTTP_CLIENT=OFF is not drop-in: it removes the only built-in HTTP client, so the consumer must supply their own IHttpClient via CFG_MODULE_HTTP_CLIENT or telemetry silently cannot upload. Shipping that as a published-port feature would let a consumer enable it and get a port that compiles but fails to send data at runtime, with no build-time signal.
BUILD_CURL_HTTP_CLIENT therefore stays a CMake build option for direct-source embedders (e.g. ONNX Runtime / Foundry Local, which build the SDK via CMake/FetchContent and already provide their own transport), where the host owns the integration. The vcpkg port keeps a working default HTTP client (and its curl dependency) so find_package(MSTelemetry) consumers get a functional SDK out of the box.
The comment-accuracy point (line 490) is addressed in 534be72.
Correct the inline comment: a PRIVATE link of the bundled SQLite suppresses propagation of its include dirs / compile definitions, but a static mat still propagates the archive for linking via \$<LINK_ONLY:...> (hence it is exported for static builds); a shared mat absorbs it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Two opt-in build options that reduce the SDK's binary footprint when it is embedded into a host application (e.g. ONNX Runtime / Foundry Local). Both default OFF, so existing builds are unchanged.
1.
BUILD_CURL_HTTP_CLIENT(default ON) — build without curl/TLSOn Linux/Android, the default HTTP client links libcurl, which transitively pulls in a TLS backend (OpenSSL). Hosts that already own transport (and supply their own
IHttpClientviaCFG_MODULE_HTTP_CLIENT) pay for an HTTP stack they never use.Setting
BUILD_CURL_HTTP_CLIENT=OFFdefines-DMATSDK_NO_DEFAULT_HTTP_CLIENT, whichlib/include/mat/config.huses to undefineHAVE_MAT_DEFAULT_HTTP_CLIENT. The curl factory then compiles to nothing andLogManagerImplrequires a host-supplied client (it fails closed —throw std::invalid_argumentif none is provided; there is no plaintext fallback). Apple (NSURLSession) and Windows (WinInet) are already curl/OpenSSL-free.2.
MATSDK_MINIMAL_SQLITE(default OFF) — private feature-stripped SQLiteThe SDK uses SQLite only for its offline event-storage cache (plain tables, transactions, WAL, autovacuum/
VACUUM, a few PRAGMAs, and one custom UTF-8 function).MATSDK_MINIMAL_SQLITE=ONcompiles a private SQLite from the vendored amalgamation with a set of amalgamation-safe strip flags (single source of truth:MATSDK_SQLITE_MINIMAL_DEFS), removing the externalsqlite3dependency and shrinking SQLite ~10.2% (.text) / ~12.5% (stripped object).minimal-sqlitefeature;sqlite3moves into a defaultsystem-sqlitefeature, so[core,minimal-sqlite]drops the external dependency.SQLITE_OMIT_AUTOINIT(theskipSqliteInitAndShutdownconfig lets the host skip the SDK's explicitsqlite3_initialize(), which the host cannot do against a private SQLite) andSQLITE_DEFAULT_MEMSTATUS=0(the SDK arms a soft heap limit viasqlite3_soft_heap_limit64()that is only enforced while memory statistics are on).matpropagates the private SQLite through its link interface, so it is exported/installed asMSTelemetry::sqlite3_bundled. The footprint doc notes the static-absorption symbol-visibility caveat.Validation
MATSDK_MINIMAL_SQLITE=ON[core,minimal-sqlite]consumerMSTelemetry::sqlite3_bundled, runs 10/10; external sqlite3 dropped; config skipsfind_dependencysystem-sqlite)sqlite3_bundled+matcompile/link with the/wpath.text/ −12.5% stripped objectReviewed for correctness, security (strip is security-neutral-to-positive —
SQLITE_DQS=0hardens identifier handling; no-curl fails closed), and design before opening.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com