Autonomous Delivery Navigation Agent

The goal was to build a navigation agent that does more than move from one point to another. It must follow pickup-and-delivery rules, avoid unsafe movement, handle penalty zones, and react to moving blockers. The project studies how reward design changes the agent’s behavior inside a constrained environment.

• The task follows a fixed sequence: start, pickup, then delivery.
• The map includes static obstacles, penalty zones, and dynamic obstacle paths.
• The main question is whether reward shaping improves route quality, waiting behavior, and completion reliability.

The final design uses a hybrid planner-guided RL approach. A* provides route structure, the DQN scores local actions, and a safety layer blocks invalid moves. This keeps learning focused on decision quality instead of basic path feasibility.

• The agent is tested inside a rule-based delivery task, not just a free-movement demo.
• Planner guidance reduces invalid rollouts while keeping learned action selection in the loop.
• The replay layer makes route quality and failure cases easier to inspect.

The workflow started with the environment, then moved into reward shaping, policy training, inference, and visual inspection. The 3D interface supports the training-to-evaluation loop by making each rollout easier to analyze.

• Each stage supports the next: environment first, then rewards, then training, then replay.
• Environment constraints directly affect learning by attaching cost to unsafe or invalid behavior.
• Backend rollout output connects to a React Three Fiber visualizer for inspection.

A pure DQN policy can be unstable on user-created layouts, so the final approach separates global route structure from local action choice. This makes execution more reliable on custom maps while still using learned behavior for movement decisions.

• The DQN still matters because it handles local action choice under reward pressure.
• The planner stabilizes execution rather than replacing the learned policy.
• This setup makes reward design easier to evaluate because failed navigation does not dominate every rollout.

The reward function was designed to push the agent toward useful behavior, not just the shortest path. It combines progress signals, task rewards, wait costs, collision penalties, and penalty-zone costs so the agent must balance speed, safety, and completion.

• Reward shaping creates more realistic behavior than shortest-path movement alone.
• Penalty zones support cost-sensitive navigation instead of a simple blocked/free map.
• Waiting is allowed for dynamic obstacles, but repeated waiting becomes expensive.
• The reward values are explicit in the environment code, making behavior easier to inspect and tune.

The training pipeline uses PyTorch DQN with replay memory, a target network, exploration scheduling, and checkpoint saving. The best validation checkpoint is used for inference when available.

• The training loop includes warm-up logic before the main DQN updates.
• The backend loads the best available checkpoint instead of always using the final checkpoint.
• Saved metrics make it easier to compare success rate, reward, route quality, and collision behavior.

The project includes custom maps and replay tools so trained behavior can be inspected visually. The interface is not just a showcase; it helps check whether the policy follows the intended task rules and reward incentives.

• Supports preset scenarios and custom maps for testing different layouts.
• Shows planned route, executed route, task phase, cargo state, and failure cases.
• Helps inspect whether reward design produces sensible local decisions.

Dynamic obstacles represent moving people, carts, or blockers inside the grid world. Their paths are validated by the backend and replayed with synchronized timeline frames.

• Obstacle paths must stay in bounds, remain connected, and avoid reserved task cells.
• The backend returns timeline data so movement stays synchronized with replay.
• The agent can wait or choose safer local actions when a blocker interrupts the route.

The FastAPI backend connects the trained policy to the frontend. It validates the map, resolves the planner route, runs the guided rollout, and returns replay-ready output.

• Uses the best available validation checkpoint when present.
• Rejects impossible or invalid layouts before rollout.
• Returns route data, task metrics, planner metadata, and timeline data for evaluation.

The stack combines PyTorch reinforcement learning, planner-guided inference, FastAPI backend services, and a lightweight 3D visualization layer for inspection.

The final result is an RL-based indoor delivery simulator where a DQN policy operates inside a constrained environment with reward shaping, planner guidance, and safety validation.

• The strongest technical value is in the RL environment and reward design, not only the UI.
• The backend exposes the trained policy through replay-ready rollout data and metrics.
• The frontend helps verify whether learned behavior matches the intended reward structure.

This is a simulator, not a production robotics stack. The environment is grid-based, and the 3D scene visualizes behavior from that grid world.

• The current task format is limited to one pickup and one delivery.
• Planner-guided DQN is used here for stable simulated decision-making, not production deployment.

This project showed me that reinforcement learning depends heavily on environment design. Planning adds structure, but the biggest improvement came from shaping incentives, penalties, and constraints so the agent learns safer and more useful behavior.

• The biggest improvement came from combining planner guidance with reward-shaped DQN behavior.
• The main takeaway is that good RL systems depend on incentives and constraints as much as on the model itself.