230 lines
14 KiB
Markdown
230 lines
14 KiB
Markdown
# Phase 11: Real-World Field Test Execution Readiness & Forensic Observability Audit Report
|
||
|
||
## 1. Executive Verdict
|
||
|
||
| Verdict Dimension | Classification / Verdict |
|
||
|---|---|
|
||
| **Executive Verdict** | **READY FOR PHYSICAL FIELD TEST** |
|
||
| **Software Validation** | **PASSED** (106 / 106 automated tests passing, 0 analyzer errors) |
|
||
| **Synthetic Replay Validation** | **PASSED** (Multi-phase 30-minute replay simulation validated) |
|
||
| **Physical Motorcycle Validation** | **PENDING** (Requires physical on-motorcycle execution by rider) |
|
||
|
||
```
|
||
+==================================================================================================+
|
||
| EXECUTIVE VERDICT |
|
||
| |
|
||
| >>> READY FOR PHYSICAL FIELD TEST <<< |
|
||
| |
|
||
| DISTINCTION NOTICE: |
|
||
| 1. Software Validation: PASSED. Telemetry ingest, sensor fusion, quality validation, bounded |
|
||
| queue logging, timing forensics, crash recovery, and export systems are verified. |
|
||
| 2. Synthetic Replay Validation: PASSED. 30-minute deterministic replay verified. |
|
||
| 3. Physical Motorcycle Validation: PENDING. The actual motorcycle ride must be performed |
|
||
| by the rider. The application provides the instrumentation necessary to evaluate that |
|
||
| physical test objectively. |
|
||
+==================================================================================================+
|
||
```
|
||
|
||
---
|
||
|
||
## 2. What Was Audited
|
||
|
||
A comprehensive forensic engineering audit was performed across all critical subsystems:
|
||
1. **Telemetry & Sensor Pipeline**: `TelemetryService`, `LocationService`, `SensorFusionService`, `CompassService`, `TelemetryValidator`.
|
||
2. **Session & Lifecycle State Machine**: `RideStateMachine`, `RecordsService`, `PerformanceTimerService`.
|
||
3. **Forensic Logging & Storage**: `FieldTestLogger`, `FieldTestExportService`, `AppDatabase` (schema version 5).
|
||
4. **Android Background & Foreground Execution**: `AndroidManifest.xml`, `ForegroundNotificationConfig`, Wakelock management, location permissions, activity lifecycle handling.
|
||
5. **Observability & Health Monitoring**: Live metrics dashboard (`FieldTestLiveLogScreen`), Pre-Ride Readiness Check (`FieldTestPreRideCheckScreen`), Post-Ride Report (`FieldTestReportScreen`).
|
||
6. **Export & Replay Subsystems**: Zip package construction (`field_test_session/`), `FieldTestReplayService`, `TelemetryReplayEngine`.
|
||
|
||
---
|
||
|
||
## 3. What Was Changed
|
||
|
||
1. **Pre-Ride Readiness Diagnostics (`FieldTestPreRideCheckScreen`)**:
|
||
- Created dedicated pre-ride readiness screen verifying 10 required criteria (GPS hardware, location permissions, live accuracy $\le 15\text{m}$, IMU streams, 20Hz telemetry heartbeat, SQLite database read/write, bounded event queue, wakelock/foreground state, ride state machine, motorcycle profile) and 3 recommended checks (battery $\ge 30\%$, sensor freshness $\le 500\text{ms}$, mount calibration).
|
||
- Added direct route `/settings/field-test-pre-ride-check` and pre-ride launch actions in Settings and Evolution views.
|
||
2. **Queue & Observability Metrics in `FieldTestLogger`**:
|
||
- Added tracking for `peakQueueDepth`, `totalEnqueued`, `totalFlushed`, `droppedEventsCount`, `oldestDroppedItem`, `newestDroppedItem`, `lastFlushDurationMs`, and `failedFlushCount`.
|
||
- Added real-time sample rate calculation (`currentSampleRateHz`) and timing jitter tracking (`averageTimingJitterMs`).
|
||
3. **Timing Forensics in `FieldTestLogger`**:
|
||
- Added timestamp regression checks: flags `timestampError` if sample timestamp or GPS fix timestamp regresses relative to previous samples.
|
||
- Added timing jitter accumulation (nominal 50ms interval).
|
||
4. **Enhanced Forensic Export Package (`FieldTestExportService`)**:
|
||
- Conformed ZIP package to the exact `field_test_session/` folder standard containing:
|
||
- `field_test_session.json` (complete structured telemetry and event bundle)
|
||
- `field_test_events.csv` (standard 15-column timeline)
|
||
- `telemetry.csv` (20Hz sensor sample trace)
|
||
- `telemetry.gpx` (GPS trace with lean/speed extensions)
|
||
- `ride_summary.json` (ride summary stats)
|
||
- `README.txt` (audit metadata header explaining versions, session IDs, sample counts, and warning/error statistics).
|
||
5. **Deterministic Offline Replay Service (`FieldTestReplayService`)**:
|
||
- Implemented deterministic replay engine capable of parsing exported JSON sessions or telemetry CSVs, streaming them through `TelemetryValidator`, and reconstructing events and quality scores.
|
||
6. **Android Foreground Service & Wakelock Configuration**:
|
||
- Added `ForegroundNotificationConfig` to `AndroidSettings` in `LocationService` with `enableWakeLock: true` and persistent notification to guarantee continuous sensor capture with the screen turned off.
|
||
7. **Live Log Observability Dashboard (`FieldTestLiveLogScreen`)**:
|
||
- Added real-time status card showing recording status, session duration, telemetry rate (Hz), GPS accuracy, GPS age, queue depth/capacity, peak queue depth, flushed events, dropped events, and live quality estimate.
|
||
|
||
---
|
||
|
||
## 4. What Was Not Changed
|
||
|
||
1. **Core Physics & Sensor Fusion Math**: Preserved existing complementary/Kalman filtering, gravity decoupling, dynamic lean angle computations, and roll-rate sanity checks.
|
||
2. **Speed & Threshold Sanity Limits**: Preserved raw sensor validity speed ceiling ($\approx 100\text{ m/s} \approx 223.7\text{ MPH}$) and strict personal-record protection gates ($< 220\text{ MPH}$).
|
||
3. **Existing UI Layouts & Screen Routing**: All existing screens (Dashboard, Map, History, Garage, Records, Settings) remain unchanged in behavior and appearance.
|
||
4. **Drift Database Integrity**: Existing tables and queries were preserved; schema version 5 remains the stable foundation.
|
||
|
||
---
|
||
|
||
## 5. Background Execution Findings
|
||
|
||
- **Android Manifest & Permissions**: `ACCESS_FINE_LOCATION`, `ACCESS_COARSE_LOCATION`, `ACCESS_BACKGROUND_LOCATION`, `FOREGROUND_SERVICE`, `FOREGROUND_SERVICE_LOCATION`, `WAKE_LOCK`, and `POST_NOTIFICATIONS` are all declared.
|
||
- **Screen-Off Resilience**: `WakelockPlus.enable()` is engaged upon ride start and `ForegroundNotificationConfig` is attached to the Android location stream.
|
||
- **Activity Destruction Safety**: Core recording state is maintained in background singleton providers (`RideStateMachine`, `TelemetryService`, `FieldTestLogger`) rather than UI widgets; activity recreation does not disrupt background recording.
|
||
|
||
---
|
||
|
||
## 6. Sensor Findings
|
||
|
||
- **GPS Quality & Age**: Freshness is tracked per-sample (`gpsAgeMs`). If GPS age exceeds 2000ms at speeds $>3\text{ MPH}$, a single `GPS_STALE` warning is emitted; upon recovery ($\le 500\text{ms}$), a single `GPS_RECOVERED` event is logged with total outage duration and peak metrics.
|
||
- **GPS Multi-Path Rejection**: Teleportation jumps ($>110\text{ m/s}$ with $>50\text{m}$ jump in $<3\text{s}$) are rejected with `GPS_JUMP_REJECTED`.
|
||
- **IMU Saturation**: Accelerometer values $>120\text{ m/s}^2$ trigger `IMU_SATURATION` error and prevent erroneous personal record creation. Recovery is automatically logged when acceleration returns to nominal dynamic range.
|
||
|
||
---
|
||
|
||
## 7. Timing Forensics Findings
|
||
|
||
- **Timestamp Hierarchy**: Sensor event time (`sample.timestamp`), GPS fix time (`sample.gpsTimestamp`), telemetry emission time (`now`), and SQLite insertion time are strictly decoupled.
|
||
- **Regression & Jitter Detection**: Samples with backward-stepping timestamps trigger `TIMESTAMP_ERROR` events. Average timing jitter relative to the nominal 50ms interval is recorded.
|
||
- **Telemetry Gaps**: Stream interruptions $>2000\text{ms}$ trigger `TELEMETRY_GAP` events with delta time, followed by `TELEMETRY_RECOVERED` upon resumption.
|
||
|
||
---
|
||
|
||
## 8. Queue & Database Observability Findings
|
||
|
||
- **Bounded Capacity**: Bounded queue strictly enforces 1,000-event ceiling.
|
||
- **Prioritized Drop Policy**: When capacity is reached, lowest-priority `info` events are evicted first to ensure `error` and `critical` events are never lost.
|
||
- **Batch Persistence**: Asynchronous non-blocking SQLite flush executes every 3 seconds in a single database transaction. Flush latency is recorded (`lastFlushDurationMs`).
|
||
|
||
---
|
||
|
||
## 9. Crash Recovery Findings
|
||
|
||
- **Interrupted Ride Identification**: On application boot, any ride in the database with `endTime == null` is detected.
|
||
- **Timestamp Reconstruction**: End time is safely calculated from the last recorded telemetry sample's timestamp (or moving time), never fabricated.
|
||
- **Forensic Summary Generation**: `generateAndSaveSummary(rideId, wasRecovered: true)` automatically computes and stores a complete quality summary marked with `wasRecoveredAfterCrash = true` and `wasInterrupted = true`.
|
||
|
||
---
|
||
|
||
## 10. Export Findings
|
||
|
||
- **Zip Package Layout**:
|
||
```
|
||
field_test_session/
|
||
├── README.txt
|
||
├── field_test_session.json
|
||
├── field_test_events.csv
|
||
├── telemetry.csv
|
||
├── telemetry.gpx
|
||
└── ride_summary.json
|
||
```
|
||
- **Integrity**: CSV formatting escapes quotes properly; JSON output is indented and valid; GPX XML includes custom telemetry extensions.
|
||
|
||
---
|
||
|
||
## 11. Performance Timer Findings
|
||
|
||
- **Sanity Gating**: 0–60 MPH and 0–100 MPH sprint timers require valid start/end speeds, valid sample counts, and high confidence ($\ge 0.85$).
|
||
- **Anti-Glitch Protection**: Runs with GPS jumps, degraded accuracy ($>15\text{m}$), or impossible acceleration ($>25\text{ m/s}^2$) are rejected and logged with `PERFORMANCE_RUN_REJECTED`.
|
||
|
||
---
|
||
|
||
## 12. Test Results
|
||
|
||
- **Total Test Suites Executed**: 10 test files across `test/`.
|
||
- **Total Tests Passed**: **106 / 106 (100% Pass Rate, 0 Failed, 0 Flaky)**.
|
||
- **Key Test Suites**:
|
||
- `test/phase11_field_test_readiness_audit_test.dart`: 6 tests covering timing regressions, queue capacity drop policies, crash recovery, export layout, and deterministic replay.
|
||
- `test/phase10_realistic_30min_replay_test.dart`: 30-minute synthetic ride replay.
|
||
- `test/telemetry_validation_test.dart`: Telemetry validator sanity gates.
|
||
- `test/sensor_fusion_physics_test.dart`: Engine specifications & sensor fusion math.
|
||
- `test/production_audit_test.dart`: Database recovery, GPX/CSV export, and performance timers.
|
||
|
||
---
|
||
|
||
## 13. Static Analysis Result
|
||
|
||
```
|
||
flutter analyze lib/ test/
|
||
Analyzing 2 items...
|
||
No issues found! (ran in 3.7s)
|
||
```
|
||
- **0 errors, 0 warnings, 0 lints**.
|
||
|
||
---
|
||
|
||
## 14. Release Build Result
|
||
|
||
```
|
||
flutter build apk --release
|
||
√ Built build\app\outputs\flutter-apk\app-release.apk (63.4MB)
|
||
```
|
||
- Clean release artifact ready for physical installation.
|
||
|
||
---
|
||
|
||
## 15. Known Limitations
|
||
|
||
1. **Barometric Altitude**: Altitude is sourced from GPS fix; high vertical accuracy is subject to GPS constellation geometry.
|
||
2. **Extreme Engine Vibration**: Mounting the phone directly to handlebars without rubber vibration damping can introduce high-frequency noise into IMU data; vibration damping mounts (e.g. Quad Lock Vibration Dampener) are strongly recommended.
|
||
3. **Zero-Interaction Intentionality**: The rider cannot manually edit or annotate logs while riding; all diagnostics are captured automatically.
|
||
|
||
---
|
||
|
||
## 16. Exact Physical Field-Test Procedure
|
||
|
||
### Phase A: Pre-Flight Preparation
|
||
1. **Charge Phone**: Ensure device battery is $\ge 50\%$.
|
||
2. **Mount Phone**: Secure phone in a vibration-damped motorcycle mount in landscape or portrait orientation.
|
||
3. **Open MotorSignal**: Launch the app and verify GPS signal (green dot / accuracy $\le 10\text{m}$).
|
||
4. **Run Pre-Ride Readiness Check**:
|
||
- Open **Settings $\to$ Pre-Ride Check** (or tap **Pre-Ride Check** button in Evolution Settings).
|
||
- Verify banner displays **`READY FOR FIELD TEST`**.
|
||
5. **Arm Field Test Mode**: Ensure **Field Test Mode** switch is **ON**.
|
||
6. **Start Ride**: Tap **Start Field Test & Begin Ride** (or **Start Ride** on Dashboard).
|
||
7. **Stow / Mount Phone**: Put phone in mounting bracket or tank bag. **Do NOT touch or interact with the phone while moving.**
|
||
|
||
### Phase B: On-Road Riding Profile (30–60 Minutes)
|
||
- **10 Minutes Urban Stop/Start**: Ride through local streets with traffic lights and stop signs (tests 0 MPH stationary wander suppression and 0–35 MPH transitions).
|
||
- **15–20 Minutes Steady Highway Cruising**: Maintain 55–75 MPH steady speed (tests GPS freshness, 20Hz heartbeat stability, and baseline vibration).
|
||
- **5–10 Minutes Moderate Cornering**: Ride through gentle and moderate curves up to $30^\circ$–$40^\circ$ lean angles (tests IMU dynamic lean filtering).
|
||
- **Normal Riding**: Complete your return trip normally.
|
||
- *Safety Rule: Do NOT perform dangerous or abrupt maneuvers solely to test telemetry. Observe all traffic laws.*
|
||
|
||
---
|
||
|
||
## 17. Exact Post-Ride Validation Procedure
|
||
|
||
1. **Stop Motorcycle Safely**: Park bike and turn off ignition.
|
||
2. **Finish Ride**: Tap **Finish Ride** on the MotorSignal screen.
|
||
3. **Allow Logger to Flush**: Wait 3–5 seconds for final SQLite transactions to settle.
|
||
4. **Open Forensic Report**:
|
||
- Navigate to **Settings $\to$ View Reports** (or select the completed ride from History).
|
||
- Review **Overall Quality Score** (target: $\ge 85 / 100$, rating: `EXCELLENT`).
|
||
- Review **Sensor Health**: Confirm GPS fix count, low stale count, and $0$ IMU saturation events.
|
||
- Review **Event Timeline**: Check for any unexpected `WARNING` or `ERROR` events.
|
||
5. **Export Forensic Bundle**:
|
||
- Tap the **Share/Export** button in the AppBar.
|
||
- Share or save the `.zip` package (`field_test_session_*.zip`).
|
||
6. **Inspect Archive Contents**:
|
||
- Confirm presence of `field_test_session.json`, `field_test_events.csv`, `telemetry.csv`, `telemetry.gpx`, `ride_summary.json`, and `README.txt`.
|
||
7. **Offline Replay Verification**:
|
||
- Feed exported telemetry to `FieldTestReplayService` to verify deterministic recreation of metrics.
|
||
|
||
---
|
||
|
||
## 18. Remaining Risks
|
||
|
||
1. **Aggressive OEM Battery Optimization**: Some Android manufacturers (e.g. Xiaomi MIUI, Huawei EMUI, Samsung OneUI) employ aggressive background task killers. Ensure MotorSignal is exempted from battery optimization ("Unrestricted" battery setting).
|
||
2. **Thermal Throttling**: Direct sunlight on a hot day with the phone mounted in a case can cause phone thermal throttling. Using an open-air mount mitigates heat buildup.
|