Settings placeholder substitution (#164)

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Configured the WiFi host name to contain the device id by default

* Removed possibility to use placeholders for FACTORY_WIFI_SSID factory setting.

* Update README.md

Updated documentation

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Configured the WiFi host name to contain the device id by default

* Removed possibility to use placeholders for FACTORY_WIFI_SSID factory setting.

* Added a space to the end of the file to comply project code style

* fix typos
clang formatting
use 2 spaces in ini files
use ${platform}-${chip_id} for hostname
put chip id in brackets in AP SSID

* restore (and update) factory setting ifndefs

- this is so src can be built without an exaustive set build-time defines
- standardize ordering of defines: factory settings, paths, config

* format and modify comment

* escape spaces in pio defines
experiment with removing $'s from our format strings (they are being substituted with empty values by pio)

* fix formatting in readme
rename FactoryValue to SettingValue, put in own header
give example of direct usage of FactorySetting::format in README.md

* auto format

* use hash to delimit placeholders

* fix factory_settings.ini

* remove flash string helpers

* format ini file

* use MAC address instead of chip id for properly unique identifier

* use lower case hex encoding for unique id
use chip id and unique id for more secure secret

* fix comment

* Use random values for JWT secret
Arduino uses the ESP random number generator for "true random" numbers on both esp32 and esp8266
This makes a better JWT secret and may be useful for other factory defaults too
In addition a modification has been made to force the FSPersistance to save the file if applying defaults

* Don't use spaces in default AP SSID

* restore helpful comment in factory_settings.ini
fix default defines

Co-authored-by: kasedy <kasedy@gmail.com>

README.md

@@ -169,12 +169,12 @@ Many of the framework's built in features may be enabled or disabled as required
 Customize the settings as you see fit. A value of 0 will disable the specified feature:
 
 ```ini
-    -D FT_PROJECT=1
+  -D FT_PROJECT=1
-    -D FT_SECURITY=1
+  -D FT_SECURITY=1
-    -D FT_MQTT=1
+  -D FT_MQTT=1
-    -D FT_NTP=1
+  -D FT_NTP=1
-    -D FT_OTA=1
+  -D FT_OTA=1
-    -D FT_UPLOAD_FIRMWARE=1
+  -D FT_UPLOAD_FIRMWARE=1
 ```
 
 Flag               | Description
@@ -193,9 +193,9 @@ The framework has built-in factory settings which act as default values for the
 Customize the settings as you see fit, for example you might configure your home WiFi network as the factory default:
 
 ```ini
-    -D FACTORY_WIFI_SSID=\"My Awesome WiFi Network\"
+  -D FACTORY_WIFI_SSID=\"My Awesome WiFi Network\"
-    -D FACTORY_WIFI_PASSWORD=\"secret\"
+  -D FACTORY_WIFI_PASSWORD=\"secret\"
-    -D FACTORY_WIFI_HOSTNAME=\"awesome_light_controller\"
+  -D FACTORY_WIFI_HOSTNAME=\"awesome_light_controller\"
 ```
 
 ### Default access point settings
@@ -221,15 +221,31 @@ It is recommended that you change the user credentials from their defaults bette
 Changing factory time zone setting is a common requirement. This requires a little effort because the time zone name and POSIX format are stored as separate values for the moment. The time zone names and POSIX formats are contained in the UI code in [TZ.tsx](interface/src/ntp/TZ.tsx). Take the appropriate pair of values from there, for example, for Los Angeles you would use:
 
 ```ini
-    -D FACTORY_NTP_TIME_ZONE_LABEL=\"America/Los_Angeles\"
+  -D FACTORY_NTP_TIME_ZONE_LABEL=\"America/Los_Angeles\"
-    -D FACTORY_NTP_TIME_ZONE_FORMAT=\"PST8PDT,M3.2.0,M11.1.0\"
+  -D FACTORY_NTP_TIME_ZONE_FORMAT=\"PST8PDT,M3.2.0,M11.1.0\"
 ```
 
-### Device ID factory defaults
+### Placeholder substitution
 
-If not overridden with a build flag, the framework will use the device ID to generate factory defaults for settings such as the JWT secret and MQTT client ID. 
+Various settings support placeholder substitution, indicated by comments in [factory_settings.ini](factory_settings.ini). This can be particularly useful where settings need to be unique, such as the Access Point SSID or MQTT client id. The following placeholders are supported:
 
-> **Tip**: Random values are generally better defaults for these settings, so it is recommended you leave these flags undefined.
+Placeholder  | Substituted value 
+-----------  | -----------------
+#{platform}  | The microcontroller platform, e.g. "esp32" or "esp8266"
+#{unique_id} | A unique identifier derived from the MAC address, e.g. "0b0a859d6816"
+#{random}    | A random number encoded as a hex string, e.g. "55722f94"
+
+You may use SettingValue::format in your own code if you require the use of these placeholders. This is demonstrated in the demo project:
+
+```cpp
+  static StateUpdateResult update(JsonObject& root, LightMqttSettings& settings) {
+    settings.mqttPath = root["mqtt_path"] | SettingValue::format("homeassistant/light/#{unique_id}");
+    settings.name = root["name"] | SettingValue::format("light-#{unique_id}");
+    settings.uniqueId = root["unique_id"] | SettingValue::format("light-#{unique_id}");
+    return StateUpdateResult::CHANGED;
+  }
+};
+```
 
 ## Building for different devices
 

factory_settings.ini

@@ -1,48 +1,51 @@
+; The indicated settings support placeholder substitution as follows:
+;
+;  #{platform} - The microcontroller platform, e.g. "esp32" or "esp8266"
+;  #{unique_id} - A unique identifier derived from the MAC address, e.g. "0b0a859d6816"
+;  #{random} - A random number encoded as a hex string, e.g. "55722f94"
+
 [factory_settings]
 build_flags =
-    ; WiFi settings
+  ; WiFi settings
-    -D FACTORY_WIFI_SSID=\"\"
+  -D FACTORY_WIFI_SSID=\"\"
-    -D FACTORY_WIFI_PASSWORD=\"\"
+  -D FACTORY_WIFI_PASSWORD=\"\"
-    ; if unspecified the devices hardware ID will be used
+  -D FACTORY_WIFI_HOSTNAME=\"#{platform}-#{unique_id}\" ; supports placeholders
-    ; -D FACTORY_WIFI_HOSTNAME=\"esp-react\"  
 
-    ; Access point settings
+  ; Access point settings
-    -D FACTORY_AP_PROVISION_MODE=AP_MODE_DISCONNECTED
+  -D FACTORY_AP_PROVISION_MODE=AP_MODE_DISCONNECTED
-    -D FACTORY_AP_SSID=\"ESP8266-React\" ; 1-64 characters
+  -D FACTORY_AP_SSID=\"ESP8266-React-#{unique_id}\" ; 1-64 characters, supports placeholders
-    -D FACTORY_AP_PASSWORD=\"esp-react\" ; 8-64 characters
+  -D FACTORY_AP_PASSWORD=\"esp-react\" ; 8-64 characters
-    -D FACTORY_AP_LOCAL_IP=\"192.168.4.1\"
+  -D FACTORY_AP_LOCAL_IP=\"192.168.4.1\"
-    -D FACTORY_AP_GATEWAY_IP=\"192.168.4.1\"
+  -D FACTORY_AP_GATEWAY_IP=\"192.168.4.1\"
-    -D FACTORY_AP_SUBNET_MASK=\"255.255.255.0\"
+  -D FACTORY_AP_SUBNET_MASK=\"255.255.255.0\"
 
-    ; User credentials for admin and guest user
+  ; User credentials for admin and guest user
-    -D FACTORY_ADMIN_USERNAME=\"admin\"
+  -D FACTORY_ADMIN_USERNAME=\"admin\"
-    -D FACTORY_ADMIN_PASSWORD=\"admin\"
+  -D FACTORY_ADMIN_PASSWORD=\"admin\"
-    -D FACTORY_GUEST_USERNAME=\"guest\"
+  -D FACTORY_GUEST_USERNAME=\"guest\"
-    -D FACTORY_GUEST_PASSWORD=\"guest\"
+  -D FACTORY_GUEST_PASSWORD=\"guest\"
 
-    ; NTP settings
+  ; NTP settings
-    -D FACTORY_NTP_ENABLED=true
+  -D FACTORY_NTP_ENABLED=true
-    -D FACTORY_NTP_TIME_ZONE_LABEL=\"Europe/London\"
+  -D FACTORY_NTP_TIME_ZONE_LABEL=\"Europe/London\"
-    -D FACTORY_NTP_TIME_ZONE_FORMAT=\"GMT0BST,M3.5.0/1,M10.5.0\"
+  -D FACTORY_NTP_TIME_ZONE_FORMAT=\"GMT0BST,M3.5.0/1,M10.5.0\"
-    -D FACTORY_NTP_SERVER=\"time.google.com\"
+  -D FACTORY_NTP_SERVER=\"time.google.com\"
 
-    ; OTA settings
+  ; OTA settings
-    -D FACTORY_OTA_PORT=8266
+  -D FACTORY_OTA_PORT=8266
-    -D FACTORY_OTA_PASSWORD=\"esp-react\"
+  -D FACTORY_OTA_PASSWORD=\"esp-react\"
-    -D FACTORY_OTA_ENABLED=true
+  -D FACTORY_OTA_ENABLED=true
 
-    ; MQTT settings
+  ; MQTT settings
-    -D FACTORY_MQTT_ENABLED=false
+  -D FACTORY_MQTT_ENABLED=false
-    -D FACTORY_MQTT_HOST=\"test.mosquitto.org\"
+  -D FACTORY_MQTT_HOST=\"test.mosquitto.org\"
-    -D FACTORY_MQTT_PORT=1883
+  -D FACTORY_MQTT_PORT=1883
-    -D FACTORY_MQTT_USERNAME=\"\"
+  -D FACTORY_MQTT_USERNAME=\"\" ; supports placeholders
-    -D FACTORY_MQTT_PASSWORD=\"\"
+  -D FACTORY_MQTT_PASSWORD=\"\"
-    ; if unspecified the devices hardware ID will be used
+  -D FACTORY_MQTT_CLIENT_ID=\"#{platform}-#{unique_id}\" ; supports placeholders
-    ;-D FACTORY_MQTT_CLIENT_ID=\"esp-react\"
+  -D FACTORY_MQTT_KEEP_ALIVE=60
-    -D FACTORY_MQTT_KEEP_ALIVE=60
+  -D FACTORY_MQTT_CLEAN_SESSION=true
-    -D FACTORY_MQTT_CLEAN_SESSION=true
+  -D FACTORY_MQTT_MAX_TOPIC_LENGTH=128
-    -D FACTORY_MQTT_MAX_TOPIC_LENGTH=128
 
-    ; JWT Secret
+  ; JWT Secret
-    ; if unspecified the devices hardware ID will be used
+  -D FACTORY_JWT_SECRET=\"#{random}-#{random}\" ; supports placeholders
-    ; -D FACTORY_JWT_SECRET=\"esp8266-react\"

features.ini

@@ -1,8 +1,8 @@
 [features]
 build_flags = 
-    -D FT_PROJECT=1
+  -D FT_PROJECT=1
-    -D FT_SECURITY=1
+  -D FT_SECURITY=1
-    -D FT_MQTT=1
+  -D FT_MQTT=1
-    -D FT_NTP=1
+  -D FT_NTP=1
-    -D FT_OTA=1
+  -D FT_OTA=1
-    -D FT_UPLOAD_FIRMWARE=1
+  -D FT_UPLOAD_FIRMWARE=1

lib/framework/APSettingsService.h

@@ -1,6 +1,7 @@
 #ifndef APSettingsConfig_h
 #define APSettingsConfig_h
 
+#include <SettingValue.h>
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 #include <JsonUtils.h>
@@ -8,20 +9,12 @@
 #include <DNSServer.h>
 #include <IPAddress.h>
 
-#define MANAGE_NETWORK_DELAY 10000
-
-#define AP_MODE_ALWAYS 0
-#define AP_MODE_DISCONNECTED 1
-#define AP_MODE_NEVER 2
-
-#define DNS_PORT 53
-
 #ifndef FACTORY_AP_PROVISION_MODE
 #define FACTORY_AP_PROVISION_MODE AP_MODE_DISCONNECTED
 #endif
 
 #ifndef FACTORY_AP_SSID
-#define FACTORY_AP_SSID "ESP8266-React"
+#define FACTORY_AP_SSID "ESP8266-React-#{unique_id}"
 #endif
 
 #ifndef FACTORY_AP_PASSWORD
@@ -43,6 +36,14 @@
 #define AP_SETTINGS_FILE "/config/apSettings.json"
 #define AP_SETTINGS_SERVICE_PATH "/rest/apSettings"
 
+#define MANAGE_NETWORK_DELAY 10000
+
+#define AP_MODE_ALWAYS 0
+#define AP_MODE_DISCONNECTED 1
+#define AP_MODE_NEVER 2
+
+#define DNS_PORT 53
+
 enum APNetworkStatus { ACTIVE = 0, INACTIVE, LINGERING };
 
 class APSettings {
@@ -79,7 +80,7 @@ class APSettings {
       default:
         newSettings.provisionMode = AP_MODE_ALWAYS;
     }
-    newSettings.ssid = root["ssid"] | FACTORY_AP_SSID;
+    newSettings.ssid = root["ssid"] | SettingValue::format(FACTORY_AP_SSID);
     newSettings.password = root["password"] | FACTORY_AP_PASSWORD;
 
     JsonUtils::readIP(root, "local_ip", newSettings.localIP, FACTORY_AP_LOCAL_IP);

lib/framework/ESPUtils.h

@@ -1,17 +0,0 @@
-#ifndef ESPUtils_h
-#define ESPUtils_h
-
-#include <Arduino.h>
-
-class ESPUtils {
- public:
-  static String defaultDeviceValue(String prefix = "") {
-#ifdef ESP32
-    return prefix + String((unsigned long)ESP.getEfuseMac(), HEX);
-#elif defined(ESP8266)
-    return prefix + String(ESP.getChipId(), HEX);
-#endif
-  }
-};
-
-#endif  // end ESPUtils

lib/framework/FSPersistence.h

@@ -38,9 +38,11 @@ class FSPersistence {
       settingsFile.close();
     }
 
-    // If we reach here we have not been successful in loading the config,
+    // If we reach here we have not been successful in loading the config and hard-coded defaults are now applied.
-    // hard-coded emergency defaults are now applied.
+    // The settings are then written back to the file system so the defaults persist between resets. This last step is
+    // required as in some cases defaults contain randomly generated values which would otherwise be modified on reset.
     applyDefaults();
+    writeToFS();
   }
 
   bool writeToFS() {

lib/framework/MqttSettingsService.h

@@ -5,12 +5,7 @@
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 #include <AsyncMqttClient.h>
-#include <ESPUtils.h>
+#include <SettingValue.h>
-
-#define MQTT_RECONNECTION_DELAY 5000
-
-#define MQTT_SETTINGS_FILE "/config/mqttSettings.json"
-#define MQTT_SETTINGS_SERVICE_PATH "/rest/mqttSettings"
 
 #ifndef FACTORY_MQTT_ENABLED
 #define FACTORY_MQTT_ENABLED false
@@ -33,7 +28,7 @@
 #endif
 
 #ifndef FACTORY_MQTT_CLIENT_ID
-#define FACTORY_MQTT_CLIENT_ID generateClientId()
+#define FACTORY_MQTT_CLIENT_ID "#{platform}-#{unique_id}"
 #endif
 
 #ifndef FACTORY_MQTT_KEEP_ALIVE
@@ -48,13 +43,10 @@
 #define FACTORY_MQTT_MAX_TOPIC_LENGTH 128
 #endif
 
-static String generateClientId() {
+#define MQTT_SETTINGS_FILE "/config/mqttSettings.json"
-#ifdef ESP32
+#define MQTT_SETTINGS_SERVICE_PATH "/rest/mqttSettings"
-  return ESPUtils::defaultDeviceValue("esp32-");
+
-#elif defined(ESP8266)
+#define MQTT_RECONNECTION_DELAY 5000
-  return ESPUtils::defaultDeviceValue("esp8266-");
-#endif
-}
 
 class MqttSettings {
  public:
@@ -91,9 +83,9 @@ class MqttSettings {
     settings.enabled = root["enabled"] | FACTORY_MQTT_ENABLED;
     settings.host = root["host"] | FACTORY_MQTT_HOST;
     settings.port = root["port"] | FACTORY_MQTT_PORT;
-    settings.username = root["username"] | FACTORY_MQTT_USERNAME;
+    settings.username = root["username"] | SettingValue::format(FACTORY_MQTT_USERNAME);
     settings.password = root["password"] | FACTORY_MQTT_PASSWORD;
-    settings.clientId = root["client_id"] | FACTORY_MQTT_CLIENT_ID;
+    settings.clientId = root["client_id"] | SettingValue::format(FACTORY_MQTT_CLIENT_ID);
     settings.keepAlive = root["keep_alive"] | FACTORY_MQTT_KEEP_ALIVE;
     settings.cleanSession = root["clean_session"] | FACTORY_MQTT_CLEAN_SESSION;
     settings.maxTopicLength = root["max_topic_length"] | FACTORY_MQTT_MAX_TOPIC_LENGTH;

lib/framework/SecurityManager.h

@@ -4,14 +4,9 @@
 #include <Features.h>
 #include <ArduinoJsonJWT.h>
 #include <ESPAsyncWebServer.h>
-#include <ESPUtils.h>
 #include <AsyncJson.h>
 #include <list>
 
-#ifndef FACTORY_JWT_SECRET
-#define FACTORY_JWT_SECRET ESPUtils::defaultDeviceValue()
-#endif
-
 #define ACCESS_TOKEN_PARAMATER "access_token"
 
 #define AUTHORIZATION_HEADER "Authorization"

lib/framework/SecuritySettingsService.h

@@ -1,11 +1,16 @@
 #ifndef SecuritySettingsService_h
 #define SecuritySettingsService_h
 
+#include <SettingValue.h>
 #include <Features.h>
 #include <SecurityManager.h>
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 
+#ifndef FACTORY_JWT_SECRET
+#define FACTORY_JWT_SECRET "#{random}-#{random}"
+#endif
+
 #ifndef FACTORY_ADMIN_USERNAME
 #define FACTORY_ADMIN_USERNAME "admin"
 #endif
@@ -48,7 +53,7 @@ class SecuritySettings {
 
   static StateUpdateResult update(JsonObject& root, SecuritySettings& settings) {
     // secret
-    settings.jwtSecret = root["jwt_secret"] | FACTORY_JWT_SECRET;
+    settings.jwtSecret = root["jwt_secret"] | SettingValue::format(FACTORY_JWT_SECRET);
 
     // users
     settings.users.clear();

lib/framework/SettingValue.cpp

@@ -0,0 +1,55 @@
+#include <SettingValue.h>
+
+namespace SettingValue {
+
+#ifdef ESP32
+const String PLATFORM = "esp32";
+#elif defined(ESP8266)
+const String PLATFORM = "esp8266";
+#endif
+
+/**
+ * Returns a new string after replacing each instance of the pattern with a value generated by calling the provided
+ * callback.
+ */
+String replaceEach(String value, String pattern, String (*generateReplacement)()) {
+  while (true) {
+    int index = value.indexOf(pattern);
+    if (index == -1) {
+      break;
+    }
+    value = value.substring(0, index) + generateReplacement() + value.substring(index + pattern.length());
+  }
+  return value;
+}
+
+/**
+ * Generates a random number, encoded as a hex string.
+ */
+String getRandom() {
+  return String(random(2147483647), HEX);
+}
+
+/**
+ * Uses the station's MAC address to create a unique id for each device.
+ */
+String getUniqueId() {
+  uint8_t mac[6];
+#ifdef ESP32
+  esp_read_mac(mac, ESP_MAC_WIFI_STA);
+#elif defined(ESP8266)
+  wifi_get_macaddr(STATION_IF, mac);
+#endif
+  char macStr[13] = {0};
+  sprintf(macStr, "%02x%02x%02x%02x%02x%02x", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+  return String(macStr);
+}
+
+String format(String value) {
+  value = replaceEach(value, "#{random}", getRandom);
+  value.replace("#{unique_id}", getUniqueId());
+  value.replace("#{platform}", PLATFORM);
+  return value;
+}
+
+};  // end namespace SettingValue

lib/framework/SettingValue.h

@@ -0,0 +1,14 @@
+#ifndef SettingValue_h
+#define SettingValue_h
+
+#include <Arduino.h>
+
+#ifdef ESP8266
+#include <ESP8266WiFi.h>
+#endif
+
+namespace SettingValue {
+String format(String value);
+};
+
+#endif  // end SettingValue

lib/framework/WiFiSettingsService.h

@@ -1,15 +1,12 @@
 #ifndef WiFiSettingsService_h
 #define WiFiSettingsService_h
 
+#include <SettingValue.h>
 #include <StatefulService.h>
 #include <FSPersistence.h>
 #include <HttpEndpoint.h>
 #include <JsonUtils.h>
 
-#define WIFI_SETTINGS_FILE "/config/wifiSettings.json"
-#define WIFI_SETTINGS_SERVICE_PATH "/rest/wifiSettings"
-#define WIFI_RECONNECTION_DELAY 1000 * 30
-
 #ifndef FACTORY_WIFI_SSID
 #define FACTORY_WIFI_SSID ""
 #endif
@@ -19,9 +16,14 @@
 #endif
 
 #ifndef FACTORY_WIFI_HOSTNAME
-#define FACTORY_WIFI_HOSTNAME ESPUtils::defaultDeviceValue("esp-react-")
+#define FACTORY_WIFI_HOSTNAME "#{platform}-#{unique_id}"
 #endif
 
+#define WIFI_SETTINGS_FILE "/config/wifiSettings.json"
+#define WIFI_SETTINGS_SERVICE_PATH "/rest/wifiSettings"
+
+#define WIFI_RECONNECTION_DELAY 1000 * 30
+
 class WiFiSettings {
  public:
   // core wifi configuration
@@ -55,7 +57,7 @@ class WiFiSettings {
   static StateUpdateResult update(JsonObject& root, WiFiSettings& settings) {
     settings.ssid = root["ssid"] | FACTORY_WIFI_SSID;
     settings.password = root["password"] | FACTORY_WIFI_PASSWORD;
-    settings.hostname = root["hostname"] | FACTORY_WIFI_HOSTNAME;
+    settings.hostname = root["hostname"] | SettingValue::format(FACTORY_WIFI_HOSTNAME);
     settings.staticIPConfig = root["static_ip_config"] | false;
 
     // extended settings

src/LightMqttSettingsService.h

@@ -3,7 +3,7 @@
 
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
-#include <ESPUtils.h>
+#include <SettingValue.h>
 
 #define LIGHT_BROKER_SETTINGS_FILE "/config/brokerSettings.json"
 #define LIGHT_BROKER_SETTINGS_PATH "/rest/brokerSettings"
@@ -21,9 +21,9 @@ class LightMqttSettings {
   }
 
   static StateUpdateResult update(JsonObject& root, LightMqttSettings& settings) {
-    settings.mqttPath = root["mqtt_path"] | ESPUtils::defaultDeviceValue("homeassistant/light/");
+    settings.mqttPath = root["mqtt_path"] | SettingValue::format("homeassistant/light/#{unique_id}");
-    settings.name = root["name"] | ESPUtils::defaultDeviceValue("light-");
+    settings.name = root["name"] | SettingValue::format("light-#{unique_id}");
-    settings.uniqueId = root["unique_id"] | ESPUtils::defaultDeviceValue("light-");
+    settings.uniqueId = root["unique_id"] | SettingValue::format("light-#{unique_id}");
     return StateUpdateResult::CHANGED;
   }
 };