Settings placeholder substitution (#164)

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Configured the WiFi host name to contain the device id by default

* Removed possibility to use placeholders for FACTORY_WIFI_SSID factory setting.

* Update README.md

Updated documentation

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Configured the WiFi host name to contain the device id by default

* Removed possibility to use placeholders for FACTORY_WIFI_SSID factory setting.

* Added a space to the end of the file to comply project code style

* fix typos
clang formatting
use 2 spaces in ini files
use ${platform}-${chip_id} for hostname
put chip id in brackets in AP SSID

* restore (and update) factory setting ifndefs

- this is so src can be built without an exaustive set build-time defines
- standardize ordering of defines: factory settings, paths, config

* format and modify comment

* escape spaces in pio defines
experiment with removing $'s from our format strings (they are being substituted with empty values by pio)

* fix formatting in readme
rename FactoryValue to SettingValue, put in own header
give example of direct usage of FactorySetting::format in README.md

* auto format

* use hash to delimit placeholders

* fix factory_settings.ini

* remove flash string helpers

* format ini file

* use MAC address instead of chip id for properly unique identifier

* use lower case hex encoding for unique id
use chip id and unique id for more secure secret

* fix comment

* Use random values for JWT secret
Arduino uses the ESP random number generator for "true random" numbers on both esp32 and esp8266
This makes a better JWT secret and may be useful for other factory defaults too
In addition a modification has been made to force the FSPersistance to save the file if applying defaults

* Don't use spaces in default AP SSID

* restore helpful comment in factory_settings.ini
fix default defines

Co-authored-by: kasedy <kasedy@gmail.com>

README.md

@@ -225,11 +225,27 @@ Changing factory time zone setting is a common requirement. This requires a litt
   -D FACTORY_NTP_TIME_ZONE_FORMAT=\"PST8PDT,M3.2.0,M11.1.0\"
 ```
 
-### Device ID factory defaults
+### Placeholder substitution
 
-If not overridden with a build flag, the framework will use the device ID to generate factory defaults for settings such as the JWT secret and MQTT client ID. 
+Various settings support placeholder substitution, indicated by comments in [factory_settings.ini](factory_settings.ini). This can be particularly useful where settings need to be unique, such as the Access Point SSID or MQTT client id. The following placeholders are supported:
 
-> **Tip**: Random values are generally better defaults for these settings, so it is recommended you leave these flags undefined.
+Placeholder  | Substituted value 
+-----------  | -----------------
+#{platform}  | The microcontroller platform, e.g. "esp32" or "esp8266"
+#{unique_id} | A unique identifier derived from the MAC address, e.g. "0b0a859d6816"
+#{random}    | A random number encoded as a hex string, e.g. "55722f94"
+
+You may use SettingValue::format in your own code if you require the use of these placeholders. This is demonstrated in the demo project:
+
+```cpp
+  static StateUpdateResult update(JsonObject& root, LightMqttSettings& settings) {
+    settings.mqttPath = root["mqtt_path"] | SettingValue::format("homeassistant/light/#{unique_id}");
+    settings.name = root["name"] | SettingValue::format("light-#{unique_id}");
+    settings.uniqueId = root["unique_id"] | SettingValue::format("light-#{unique_id}");
+    return StateUpdateResult::CHANGED;
+  }
+};
+```
 
 ## Building for different devices
 

factory_settings.ini

@@ -1,14 +1,19 @@
+; The indicated settings support placeholder substitution as follows:
+;
+;  #{platform} - The microcontroller platform, e.g. "esp32" or "esp8266"
+;  #{unique_id} - A unique identifier derived from the MAC address, e.g. "0b0a859d6816"
+;  #{random} - A random number encoded as a hex string, e.g. "55722f94"
+
 [factory_settings]
 build_flags =
   ; WiFi settings
   -D FACTORY_WIFI_SSID=\"\"
   -D FACTORY_WIFI_PASSWORD=\"\"
-    ; if unspecified the devices hardware ID will be used
+  -D FACTORY_WIFI_HOSTNAME=\"#{platform}-#{unique_id}\" ; supports placeholders
-    ; -D FACTORY_WIFI_HOSTNAME=\"esp-react\"  
 
   ; Access point settings
   -D FACTORY_AP_PROVISION_MODE=AP_MODE_DISCONNECTED
-    -D FACTORY_AP_SSID=\"ESP8266-React\" ; 1-64 characters
+  -D FACTORY_AP_SSID=\"ESP8266-React-#{unique_id}\" ; 1-64 characters, supports placeholders
   -D FACTORY_AP_PASSWORD=\"esp-react\" ; 8-64 characters
   -D FACTORY_AP_LOCAL_IP=\"192.168.4.1\"
   -D FACTORY_AP_GATEWAY_IP=\"192.168.4.1\"
@@ -35,14 +40,12 @@ build_flags =
   -D FACTORY_MQTT_ENABLED=false
   -D FACTORY_MQTT_HOST=\"test.mosquitto.org\"
   -D FACTORY_MQTT_PORT=1883
-    -D FACTORY_MQTT_USERNAME=\"\"
+  -D FACTORY_MQTT_USERNAME=\"\" ; supports placeholders
   -D FACTORY_MQTT_PASSWORD=\"\"
-    ; if unspecified the devices hardware ID will be used
+  -D FACTORY_MQTT_CLIENT_ID=\"#{platform}-#{unique_id}\" ; supports placeholders
-    ;-D FACTORY_MQTT_CLIENT_ID=\"esp-react\"
   -D FACTORY_MQTT_KEEP_ALIVE=60
   -D FACTORY_MQTT_CLEAN_SESSION=true
   -D FACTORY_MQTT_MAX_TOPIC_LENGTH=128
 
   ; JWT Secret
-    ; if unspecified the devices hardware ID will be used
+  -D FACTORY_JWT_SECRET=\"#{random}-#{random}\" ; supports placeholders
-    ; -D FACTORY_JWT_SECRET=\"esp8266-react\"

lib/framework/APSettingsService.h

@@ -1,6 +1,7 @@
 #ifndef APSettingsConfig_h
 #define APSettingsConfig_h
 
+#include <SettingValue.h>
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 #include <JsonUtils.h>
@@ -8,20 +9,12 @@
 #include <DNSServer.h>
 #include <IPAddress.h>
 
-#define MANAGE_NETWORK_DELAY 10000
-
-#define AP_MODE_ALWAYS 0
-#define AP_MODE_DISCONNECTED 1
-#define AP_MODE_NEVER 2
-
-#define DNS_PORT 53
-
 #ifndef FACTORY_AP_PROVISION_MODE
 #define FACTORY_AP_PROVISION_MODE AP_MODE_DISCONNECTED
 #endif
 
 #ifndef FACTORY_AP_SSID
-#define FACTORY_AP_SSID "ESP8266-React"
+#define FACTORY_AP_SSID "ESP8266-React-#{unique_id}"
 #endif
 
 #ifndef FACTORY_AP_PASSWORD
@@ -43,6 +36,14 @@
 #define AP_SETTINGS_FILE "/config/apSettings.json"
 #define AP_SETTINGS_SERVICE_PATH "/rest/apSettings"
 
+#define MANAGE_NETWORK_DELAY 10000
+
+#define AP_MODE_ALWAYS 0
+#define AP_MODE_DISCONNECTED 1
+#define AP_MODE_NEVER 2
+
+#define DNS_PORT 53
+
 enum APNetworkStatus { ACTIVE = 0, INACTIVE, LINGERING };
 
 class APSettings {
@@ -79,7 +80,7 @@ class APSettings {
       default:
         newSettings.provisionMode = AP_MODE_ALWAYS;
     }
-    newSettings.ssid = root["ssid"] | FACTORY_AP_SSID;
+    newSettings.ssid = root["ssid"] | SettingValue::format(FACTORY_AP_SSID);
     newSettings.password = root["password"] | FACTORY_AP_PASSWORD;
 
     JsonUtils::readIP(root, "local_ip", newSettings.localIP, FACTORY_AP_LOCAL_IP);

lib/framework/ESPUtils.h

@@ -1,17 +0,0 @@
-#ifndef ESPUtils_h
-#define ESPUtils_h
-
-#include <Arduino.h>
-
-class ESPUtils {
- public:
-  static String defaultDeviceValue(String prefix = "") {
-#ifdef ESP32
-    return prefix + String((unsigned long)ESP.getEfuseMac(), HEX);
-#elif defined(ESP8266)
-    return prefix + String(ESP.getChipId(), HEX);
-#endif
-  }
-};
-
-#endif  // end ESPUtils

lib/framework/FSPersistence.h

@@ -38,9 +38,11 @@ class FSPersistence {
       settingsFile.close();
     }
 
-    // If we reach here we have not been successful in loading the config,
+    // If we reach here we have not been successful in loading the config and hard-coded defaults are now applied.
-    // hard-coded emergency defaults are now applied.
+    // The settings are then written back to the file system so the defaults persist between resets. This last step is
+    // required as in some cases defaults contain randomly generated values which would otherwise be modified on reset.
     applyDefaults();
+    writeToFS();
   }
 
   bool writeToFS() {

lib/framework/MqttSettingsService.h

@@ -5,12 +5,7 @@
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 #include <AsyncMqttClient.h>
-#include <ESPUtils.h>
+#include <SettingValue.h>
-
-#define MQTT_RECONNECTION_DELAY 5000
-
-#define MQTT_SETTINGS_FILE "/config/mqttSettings.json"
-#define MQTT_SETTINGS_SERVICE_PATH "/rest/mqttSettings"
 
 #ifndef FACTORY_MQTT_ENABLED
 #define FACTORY_MQTT_ENABLED false
@@ -33,7 +28,7 @@
 #endif
 
 #ifndef FACTORY_MQTT_CLIENT_ID
-#define FACTORY_MQTT_CLIENT_ID generateClientId()
+#define FACTORY_MQTT_CLIENT_ID "#{platform}-#{unique_id}"
 #endif
 
 #ifndef FACTORY_MQTT_KEEP_ALIVE
@@ -48,13 +43,10 @@
 #define FACTORY_MQTT_MAX_TOPIC_LENGTH 128
 #endif
 
-static String generateClientId() {
+#define MQTT_SETTINGS_FILE "/config/mqttSettings.json"
-#ifdef ESP32
+#define MQTT_SETTINGS_SERVICE_PATH "/rest/mqttSettings"
-  return ESPUtils::defaultDeviceValue("esp32-");
+
-#elif defined(ESP8266)
+#define MQTT_RECONNECTION_DELAY 5000
-  return ESPUtils::defaultDeviceValue("esp8266-");
-#endif
-}
 
 class MqttSettings {
  public:
@@ -91,9 +83,9 @@ class MqttSettings {
     settings.enabled = root["enabled"] | FACTORY_MQTT_ENABLED;
     settings.host = root["host"] | FACTORY_MQTT_HOST;
     settings.port = root["port"] | FACTORY_MQTT_PORT;
-    settings.username = root["username"] | FACTORY_MQTT_USERNAME;
+    settings.username = root["username"] | SettingValue::format(FACTORY_MQTT_USERNAME);
     settings.password = root["password"] | FACTORY_MQTT_PASSWORD;
-    settings.clientId = root["client_id"] | FACTORY_MQTT_CLIENT_ID;
+    settings.clientId = root["client_id"] | SettingValue::format(FACTORY_MQTT_CLIENT_ID);
     settings.keepAlive = root["keep_alive"] | FACTORY_MQTT_KEEP_ALIVE;
     settings.cleanSession = root["clean_session"] | FACTORY_MQTT_CLEAN_SESSION;
     settings.maxTopicLength = root["max_topic_length"] | FACTORY_MQTT_MAX_TOPIC_LENGTH;

lib/framework/SecurityManager.h

@@ -4,14 +4,9 @@
 #include <Features.h>
 #include <ArduinoJsonJWT.h>
 #include <ESPAsyncWebServer.h>
-#include <ESPUtils.h>
 #include <AsyncJson.h>
 #include <list>
 
-#ifndef FACTORY_JWT_SECRET
-#define FACTORY_JWT_SECRET ESPUtils::defaultDeviceValue()
-#endif
-
 #define ACCESS_TOKEN_PARAMATER "access_token"
 
 #define AUTHORIZATION_HEADER "Authorization"

lib/framework/SecuritySettingsService.h

@@ -1,11 +1,16 @@
 #ifndef SecuritySettingsService_h
 #define SecuritySettingsService_h
 
+#include <SettingValue.h>
 #include <Features.h>
 #include <SecurityManager.h>
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 
+#ifndef FACTORY_JWT_SECRET
+#define FACTORY_JWT_SECRET "#{random}-#{random}"
+#endif
+
 #ifndef FACTORY_ADMIN_USERNAME
 #define FACTORY_ADMIN_USERNAME "admin"
 #endif
@@ -48,7 +53,7 @@ class SecuritySettings {
 
   static StateUpdateResult update(JsonObject& root, SecuritySettings& settings) {
     // secret
-    settings.jwtSecret = root["jwt_secret"] | FACTORY_JWT_SECRET;
+    settings.jwtSecret = root["jwt_secret"] | SettingValue::format(FACTORY_JWT_SECRET);
 
     // users
     settings.users.clear();

lib/framework/SettingValue.cpp

@@ -0,0 +1,55 @@
+#include <SettingValue.h>
+
+namespace SettingValue {
+
+#ifdef ESP32
+const String PLATFORM = "esp32";
+#elif defined(ESP8266)
+const String PLATFORM = "esp8266";
+#endif
+
+/**
+ * Returns a new string after replacing each instance of the pattern with a value generated by calling the provided
+ * callback.
+ */
+String replaceEach(String value, String pattern, String (*generateReplacement)()) {
+  while (true) {
+    int index = value.indexOf(pattern);
+    if (index == -1) {
+      break;
+    }
+    value = value.substring(0, index) + generateReplacement() + value.substring(index + pattern.length());
+  }
+  return value;
+}
+
+/**
+ * Generates a random number, encoded as a hex string.
+ */
+String getRandom() {
+  return String(random(2147483647), HEX);
+}
+
+/**
+ * Uses the station's MAC address to create a unique id for each device.
+ */
+String getUniqueId() {
+  uint8_t mac[6];
+#ifdef ESP32
+  esp_read_mac(mac, ESP_MAC_WIFI_STA);
+#elif defined(ESP8266)
+  wifi_get_macaddr(STATION_IF, mac);
+#endif
+  char macStr[13] = {0};
+  sprintf(macStr, "%02x%02x%02x%02x%02x%02x", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
+  return String(macStr);
+}
+
+String format(String value) {
+  value = replaceEach(value, "#{random}", getRandom);
+  value.replace("#{unique_id}", getUniqueId());
+  value.replace("#{platform}", PLATFORM);
+  return value;
+}
+
+};  // end namespace SettingValue

lib/framework/SettingValue.h

@@ -0,0 +1,14 @@
+#ifndef SettingValue_h
+#define SettingValue_h
+
+#include <Arduino.h>
+
+#ifdef ESP8266
+#include <ESP8266WiFi.h>
+#endif
+
+namespace SettingValue {
+String format(String value);
+};
+
+#endif  // end SettingValue

lib/framework/WiFiSettingsService.h

@@ -1,15 +1,12 @@
 #ifndef WiFiSettingsService_h
 #define WiFiSettingsService_h
 
+#include <SettingValue.h>
 #include <StatefulService.h>
 #include <FSPersistence.h>
 #include <HttpEndpoint.h>
 #include <JsonUtils.h>
 
-#define WIFI_SETTINGS_FILE "/config/wifiSettings.json"
-#define WIFI_SETTINGS_SERVICE_PATH "/rest/wifiSettings"
-#define WIFI_RECONNECTION_DELAY 1000 * 30
-
 #ifndef FACTORY_WIFI_SSID
 #define FACTORY_WIFI_SSID ""
 #endif
@@ -19,9 +16,14 @@
 #endif
 
 #ifndef FACTORY_WIFI_HOSTNAME
-#define FACTORY_WIFI_HOSTNAME ESPUtils::defaultDeviceValue("esp-react-")
+#define FACTORY_WIFI_HOSTNAME "#{platform}-#{unique_id}"
 #endif
 
+#define WIFI_SETTINGS_FILE "/config/wifiSettings.json"
+#define WIFI_SETTINGS_SERVICE_PATH "/rest/wifiSettings"
+
+#define WIFI_RECONNECTION_DELAY 1000 * 30
+
 class WiFiSettings {
  public:
   // core wifi configuration
@@ -55,7 +57,7 @@ class WiFiSettings {
   static StateUpdateResult update(JsonObject& root, WiFiSettings& settings) {
     settings.ssid = root["ssid"] | FACTORY_WIFI_SSID;
     settings.password = root["password"] | FACTORY_WIFI_PASSWORD;
-    settings.hostname = root["hostname"] | FACTORY_WIFI_HOSTNAME;
+    settings.hostname = root["hostname"] | SettingValue::format(FACTORY_WIFI_HOSTNAME);
     settings.staticIPConfig = root["static_ip_config"] | false;
 
     // extended settings

src/LightMqttSettingsService.h

@@ -3,7 +3,7 @@
 
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
-#include <ESPUtils.h>
+#include <SettingValue.h>
 
 #define LIGHT_BROKER_SETTINGS_FILE "/config/brokerSettings.json"
 #define LIGHT_BROKER_SETTINGS_PATH "/rest/brokerSettings"
@@ -21,9 +21,9 @@ class LightMqttSettings {
   }
 
   static StateUpdateResult update(JsonObject& root, LightMqttSettings& settings) {
-    settings.mqttPath = root["mqtt_path"] | ESPUtils::defaultDeviceValue("homeassistant/light/");
+    settings.mqttPath = root["mqtt_path"] | SettingValue::format("homeassistant/light/#{unique_id}");
-    settings.name = root["name"] | ESPUtils::defaultDeviceValue("light-");
+    settings.name = root["name"] | SettingValue::format("light-#{unique_id}");
-    settings.uniqueId = root["unique_id"] | ESPUtils::defaultDeviceValue("light-");
+    settings.uniqueId = root["unique_id"] | SettingValue::format("light-#{unique_id}");
     return StateUpdateResult::CHANGED;
   }
 };