motorsignal
/
MotorSignal-App
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
MotorSignal-App/PHASE11_FIELD_TEST_READINES...

14 KiB
Raw Blame History

Phase 11: Real-World Field Test Execution Readiness & Forensic Observability Audit Report

1. Executive Verdict

Verdict Dimension Classification / Verdict
Executive Verdict READY FOR PHYSICAL FIELD TEST
Software Validation PASSED (106 / 106 automated tests passing, 0 analyzer errors)
Synthetic Replay Validation PASSED (Multi-phase 30-minute replay simulation validated)
Physical Motorcycle Validation PENDING (Requires physical on-motorcycle execution by rider) 
+==================================================================================================+
|                                    EXECUTIVE VERDICT                                             |
|                                                                                                  |
|                      >>> READY FOR PHYSICAL FIELD TEST <<<                                       |
|                                                                                                  |
|  DISTINCTION NOTICE:                                                                             |
|  1. Software Validation: PASSED. Telemetry ingest, sensor fusion, quality validation, bounded    |
|     queue logging, timing forensics, crash recovery, and export systems are verified.          |
|  2. Synthetic Replay Validation: PASSED. 30-minute deterministic replay verified.                |
|  3. Physical Motorcycle Validation: PENDING. The actual motorcycle ride must be performed      |
|     by the rider. The application provides the instrumentation necessary to evaluate that        |
|     physical test objectively.                                                                   |
+==================================================================================================+

2. What Was Audited

A comprehensive forensic engineering audit was performed across all critical subsystems:

  1. Telemetry & Sensor Pipeline: TelemetryService, LocationService, SensorFusionService, CompassService, TelemetryValidator.
  2. Session & Lifecycle State Machine: RideStateMachine, RecordsService, PerformanceTimerService.
  3. Forensic Logging & Storage: FieldTestLogger, FieldTestExportService, AppDatabase (schema version 5).
  4. Android Background & Foreground Execution: AndroidManifest.xml, ForegroundNotificationConfig, Wakelock management, location permissions, activity lifecycle handling.
  5. Observability & Health Monitoring: Live metrics dashboard (FieldTestLiveLogScreen), Pre-Ride Readiness Check (FieldTestPreRideCheckScreen), Post-Ride Report (FieldTestReportScreen).
  6. Export & Replay Subsystems: Zip package construction (field_test_session/), FieldTestReplayService, TelemetryReplayEngine.

3. What Was Changed

  1. Pre-Ride Readiness Diagnostics (FieldTestPreRideCheckScreen):
    • Created dedicated pre-ride readiness screen verifying 10 required criteria (GPS hardware, location permissions, live accuracy \le 15\text{m}, IMU streams, 20Hz telemetry heartbeat, SQLite database read/write, bounded event queue, wakelock/foreground state, ride state machine, motorcycle profile) and 3 recommended checks (battery \ge 30\%, sensor freshness \le 500\text{ms}, mount calibration).
    • Added direct route /settings/field-test-pre-ride-check and pre-ride launch actions in Settings and Evolution views.
  2. Queue & Observability Metrics in FieldTestLogger:
    • Added tracking for peakQueueDepth, totalEnqueued, totalFlushed, droppedEventsCount, oldestDroppedItem, newestDroppedItem, lastFlushDurationMs, and failedFlushCount.
    • Added real-time sample rate calculation (currentSampleRateHz) and timing jitter tracking (averageTimingJitterMs).
  3. Timing Forensics in FieldTestLogger:
    • Added timestamp regression checks: flags timestampError if sample timestamp or GPS fix timestamp regresses relative to previous samples.
    • Added timing jitter accumulation (nominal 50ms interval).
  4. Enhanced Forensic Export Package (FieldTestExportService):
    • Conformed ZIP package to the exact field_test_session/ folder standard containing:
      • field_test_session.json (complete structured telemetry and event bundle)
      • field_test_events.csv (standard 15-column timeline)
      • telemetry.csv (20Hz sensor sample trace)
      • telemetry.gpx (GPS trace with lean/speed extensions)
      • ride_summary.json (ride summary stats)
      • README.txt (audit metadata header explaining versions, session IDs, sample counts, and warning/error statistics).
  5. Deterministic Offline Replay Service (FieldTestReplayService):
    • Implemented deterministic replay engine capable of parsing exported JSON sessions or telemetry CSVs, streaming them through TelemetryValidator, and reconstructing events and quality scores.
  6. Android Foreground Service & Wakelock Configuration:
    • Added ForegroundNotificationConfig to AndroidSettings in LocationService with enableWakeLock: true and persistent notification to guarantee continuous sensor capture with the screen turned off.
  7. Live Log Observability Dashboard (FieldTestLiveLogScreen):
    • Added real-time status card showing recording status, session duration, telemetry rate (Hz), GPS accuracy, GPS age, queue depth/capacity, peak queue depth, flushed events, dropped events, and live quality estimate.

4. What Was Not Changed

  1. Core Physics & Sensor Fusion Math: Preserved existing complementary/Kalman filtering, gravity decoupling, dynamic lean angle computations, and roll-rate sanity checks.
  2. Speed & Threshold Sanity Limits: Preserved raw sensor validity speed ceiling (\approx 100\text{ m/s} \approx 223.7\text{ MPH}) and strict personal-record protection gates (< 220\text{ MPH}).
  3. Existing UI Layouts & Screen Routing: All existing screens (Dashboard, Map, History, Garage, Records, Settings) remain unchanged in behavior and appearance.
  4. Drift Database Integrity: Existing tables and queries were preserved; schema version 5 remains the stable foundation.

5. Background Execution Findings

  • Android Manifest & Permissions: ACCESS_FINE_LOCATION, ACCESS_COARSE_LOCATION, ACCESS_BACKGROUND_LOCATION, FOREGROUND_SERVICE, FOREGROUND_SERVICE_LOCATION, WAKE_LOCK, and POST_NOTIFICATIONS are all declared.
  • Screen-Off Resilience: WakelockPlus.enable() is engaged upon ride start and ForegroundNotificationConfig is attached to the Android location stream.
  • Activity Destruction Safety: Core recording state is maintained in background singleton providers (RideStateMachine, TelemetryService, FieldTestLogger) rather than UI widgets; activity recreation does not disrupt background recording.

6. Sensor Findings

  • GPS Quality & Age: Freshness is tracked per-sample (gpsAgeMs). If GPS age exceeds 2000ms at speeds >3\text{ MPH}, a single GPS_STALE warning is emitted; upon recovery (\le 500\text{ms}), a single GPS_RECOVERED event is logged with total outage duration and peak metrics.
  • GPS Multi-Path Rejection: Teleportation jumps (>110\text{ m/s} with >50\text{m} jump in <3\text{s}) are rejected with GPS_JUMP_REJECTED.
  • IMU Saturation: Accelerometer values >120\text{ m/s}^2 trigger IMU_SATURATION error and prevent erroneous personal record creation. Recovery is automatically logged when acceleration returns to nominal dynamic range.

7. Timing Forensics Findings

  • Timestamp Hierarchy: Sensor event time (sample.timestamp), GPS fix time (sample.gpsTimestamp), telemetry emission time (now), and SQLite insertion time are strictly decoupled.
  • Regression & Jitter Detection: Samples with backward-stepping timestamps trigger TIMESTAMP_ERROR events. Average timing jitter relative to the nominal 50ms interval is recorded.
  • Telemetry Gaps: Stream interruptions >2000\text{ms} trigger TELEMETRY_GAP events with delta time, followed by TELEMETRY_RECOVERED upon resumption.

8. Queue & Database Observability Findings

  • Bounded Capacity: Bounded queue strictly enforces 1,000-event ceiling.
  • Prioritized Drop Policy: When capacity is reached, lowest-priority info events are evicted first to ensure error and critical events are never lost.
  • Batch Persistence: Asynchronous non-blocking SQLite flush executes every 3 seconds in a single database transaction. Flush latency is recorded (lastFlushDurationMs).

9. Crash Recovery Findings

  • Interrupted Ride Identification: On application boot, any ride in the database with endTime == null is detected.
  • Timestamp Reconstruction: End time is safely calculated from the last recorded telemetry sample's timestamp (or moving time), never fabricated.
  • Forensic Summary Generation: generateAndSaveSummary(rideId, wasRecovered: true) automatically computes and stores a complete quality summary marked with wasRecoveredAfterCrash = true and wasInterrupted = true.

10. Export Findings

  • Zip Package Layout: 
    field_test_session/
├── README.txt
├── field_test_session.json
├── field_test_events.csv
├── telemetry.csv
├── telemetry.gpx
└── ride_summary.json
  • Integrity: CSV formatting escapes quotes properly; JSON output is indented and valid; GPX XML includes custom telemetry extensions.

11. Performance Timer Findings

  • Sanity Gating: 0–60 MPH and 0–100 MPH sprint timers require valid start/end speeds, valid sample counts, and high confidence (\ge 0.85).
  • Anti-Glitch Protection: Runs with GPS jumps, degraded accuracy (>15\text{m}), or impossible acceleration (>25\text{ m/s}^2) are rejected and logged with PERFORMANCE_RUN_REJECTED.

12. Test Results

  • Total Test Suites Executed: 10 test files across test/.
  • Total Tests Passed: 106 / 106 (100% Pass Rate, 0 Failed, 0 Flaky).
  • Key Test Suites:
    • test/phase11_field_test_readiness_audit_test.dart: 6 tests covering timing regressions, queue capacity drop policies, crash recovery, export layout, and deterministic replay.
    • test/phase10_realistic_30min_replay_test.dart: 30-minute synthetic ride replay.
    • test/telemetry_validation_test.dart: Telemetry validator sanity gates.
    • test/sensor_fusion_physics_test.dart: Engine specifications & sensor fusion math.
    • test/production_audit_test.dart: Database recovery, GPX/CSV export, and performance timers.

13. Static Analysis Result

flutter analyze lib/ test/
Analyzing 2 items...
No issues found! (ran in 3.7s)
  • 0 errors, 0 warnings, 0 lints.

14. Release Build Result

flutter build apk --release
√ Built build\app\outputs\flutter-apk\app-release.apk (63.4MB)
  • Clean release artifact ready for physical installation.

15. Known Limitations

  1. Barometric Altitude: Altitude is sourced from GPS fix; high vertical accuracy is subject to GPS constellation geometry.
  2. Extreme Engine Vibration: Mounting the phone directly to handlebars without rubber vibration damping can introduce high-frequency noise into IMU data; vibration damping mounts (e.g. Quad Lock Vibration Dampener) are strongly recommended.
  3. Zero-Interaction Intentionality: The rider cannot manually edit or annotate logs while riding; all diagnostics are captured automatically.

16. Exact Physical Field-Test Procedure

Phase A: Pre-Flight Preparation

  1. Charge Phone: Ensure device battery is \ge 50\%.
  2. Mount Phone: Secure phone in a vibration-damped motorcycle mount in landscape or portrait orientation.
  3. Open MotorSignal: Launch the app and verify GPS signal (green dot / accuracy \le 10\text{m}).
  4. Run Pre-Ride Readiness Check:
    • Open Settings \to Pre-Ride Check (or tap Pre-Ride Check button in Evolution Settings).
    • Verify banner displays READY FOR FIELD TEST.
  5. Arm Field Test Mode: Ensure Field Test Mode switch is ON.
  6. Start Ride: Tap Start Field Test & Begin Ride (or Start Ride on Dashboard).
  7. Stow / Mount Phone: Put phone in mounting bracket or tank bag. Do NOT touch or interact with the phone while moving.

Phase B: On-Road Riding Profile (30–60 Minutes)

  • 10 Minutes Urban Stop/Start: Ride through local streets with traffic lights and stop signs (tests 0 MPH stationary wander suppression and 0–35 MPH transitions).
  • 15–20 Minutes Steady Highway Cruising: Maintain 55–75 MPH steady speed (tests GPS freshness, 20Hz heartbeat stability, and baseline vibration).
  • 5–10 Minutes Moderate Cornering: Ride through gentle and moderate curves up to 30^\circ–40^\circ lean angles (tests IMU dynamic lean filtering).
  • Normal Riding: Complete your return trip normally.
  • Safety Rule: Do NOT perform dangerous or abrupt maneuvers solely to test telemetry. Observe all traffic laws.

17. Exact Post-Ride Validation Procedure

  1. Stop Motorcycle Safely: Park bike and turn off ignition.
  2. Finish Ride: Tap Finish Ride on the MotorSignal screen.
  3. Allow Logger to Flush: Wait 3–5 seconds for final SQLite transactions to settle.
  4. Open Forensic Report:
    • Navigate to Settings \to View Reports (or select the completed ride from History).
    • Review Overall Quality Score (target: \ge 85 / 100, rating: EXCELLENT).
    • Review Sensor Health: Confirm GPS fix count, low stale count, and 0 IMU saturation events.
    • Review Event Timeline: Check for any unexpected WARNING or ERROR events.
  5. Export Forensic Bundle:
    • Tap the Share/Export button in the AppBar.
    • Share or save the .zip package (field_test_session_*.zip).
  6. Inspect Archive Contents:
    • Confirm presence of field_test_session.json, field_test_events.csv, telemetry.csv, telemetry.gpx, ride_summary.json, and README.txt.
  7. Offline Replay Verification:
    • Feed exported telemetry to FieldTestReplayService to verify deterministic recreation of metrics.

18. Remaining Risks

  1. Aggressive OEM Battery Optimization: Some Android manufacturers (e.g. Xiaomi MIUI, Huawei EMUI, Samsung OneUI) employ aggressive background task killers. Ensure MotorSignal is exempted from battery optimization ("Unrestricted" battery setting).
  2. Thermal Throttling: Direct sunlight on a hot day with the phone mounted in a case can cause phone thermal throttling. Using an open-air mount mitigates heat buildup.