枣强网站建设培训学校廊坊网站建设招聘

枣强网站建设培训学校,廊坊网站建设招聘,WordPress微博qq登录插件,免费建站网站一区黄在线国产强化学习项目-7-LunarLanderContinuous-v3(DDPG) 环境 本项目使用的是OpenAI Gym提供的经典控制环境的连续动作版本。与PPO笔记中的离散版本不同#xff0c;这里的动作空间是连续的数值。 官网链接#xff1a;https://gymnasium.farama.org/environments/box2d/lunar_land…强化学习项目-7-LunarLanderContinuous-v3(DDPG)环境本项目使用的是OpenAI Gym提供的经典控制环境的连续动作版本。与PPO笔记中的离散版本不同这里的动作空间是连续的数值。官网链接https://gymnasium.farama.org/environments/box2d/lunar_lander/动作空间 (Continuous)动作是一个维度为2的向量a ∈ [ − 1 , 1 ] 2 a \in [-1, 1]^2a∈[−1,1]2主引擎 (Main Engine):− 1 ∼ 0 -1 \sim 0−1∼0: 引擎关闭0 ∼ 1 0 \sim 10∼1: 引擎开启数值越大推力越大从50%到100%功率侧向推进器 (Side Engines):− 1 ∼ − 0.5 -1 \sim -0.5−1∼−0.5: 右侧推进器开启推向左− 0.5 ∼ 0.5 -0.5 \sim 0.5−0.5∼0.5: 关闭0.5 ∼ 1 0.5 \sim 10.5∼1: 左侧推进器开启推向右状态向量与离散版一致维度为8s [ x , y , x ˙ , y ˙ , θ , θ ˙ , l , r ] T s [x, y, \dot{x}, \dot{y}, \theta, \dot{\theta}, l, r]^Ts[x,y,x˙,y˙​,θ,θ˙,l,r]T奖励函数逻辑与离散版基本一致靠近平台加分、坠毁扣分等。区别连续动作版本中喷射燃料的扣分是根据动作的连续数值计算的因此更鼓励“精准控制”油门而非频繁的开关。引入环境注意需要指定continuousTrue。importgymnasiumasgym# 必须指定 continuousTrueenvgym.make(LunarLander-v3,continuousTrue,render_modehuman)state_dimenv.observation_space.shape[0]action_dimenv.action_space.shape[0]max_actionfloat(env.action_space.high[0])# 通常为 1.0DDPG 算法DDPG (Deep Deterministic Policy Gradient) 是一种基于Actor-Critic架构的算法专门用于解决连续动作空间的问题。它结合了 DQN 的思想经验回放、目标网络和确定性策略梯度。核心组件Actor 网络 (μ \muμ): 输入状态s ss直接输出确定的动作值a aa。Critic 网络 (Q QQ): 输入状态s ss和动作a aa输出该动作的价值Q ( s , a ) Q(s, a)Q(s,a)。目标网络 (Target Networks):μ ′ \muμ′和Q ′ QQ′用于计算TD目标保持训练稳定。经验回放池 (Replay Buffer): 存储( s , a , r , s ′ , d o n e ) (s, a, r, s, done)(s,a,r,s′,done)打破数据相关性。损失函数1. Critic 损失 (Value Loss)Critic 的目标是最小化预测的 Q 值与 TD Target 之间的均方误差L 1 N ∑ ( y i − Q ( s i , a i ∣ θ Q ) ) 2 L \frac{1}{N} \sum (y_i - Q(s_i, a_i|\theta^Q))^2LN1​∑(yi​−Q(si​,ai​∣θQ))2其中目标值y i y_iyi​由目标网络计算y i r i γ Q ′ ( s i 1 , μ ′ ( s i 1 ∣ θ μ ′ ) ∣ θ Q ′ ) ⋅ ( 1 − d i ) y_i r_i \gamma Q(s_{i1}, \mu(s_{i1}|\theta^{\mu})|\theta^{Q}) \cdot (1 - d_i)yi​ri​γQ′(si1​,μ′(si1​∣θμ′)∣θQ′)⋅(1−di​)2. Actor 损失 (Policy Loss)Actor 的目标是最大化 Critic 对其输出动作的评分。在梯度下降中我们通过最小化 Q 值的负数来实现J ( θ μ ) − 1 N ∑ Q ( s i , μ ( s i ∣ θ μ ) ∣ θ Q ) J(\theta^\mu) - \frac{1}{N} \sum Q(s_i, \mu(s_i|\theta^\mu)|\theta^Q)J(θμ)−N1​∑Q(si​,μ(si​∣θμ)∣θQ)探索策略 (Exploration)由于 DDPG 是确定性策略为了让智能体探索环境我们在训练时给动作添加噪声a e x e c clip ( μ ( s ) N , − a m a x , a m a x ) a_{exec} \text{clip}(\mu(s) \mathcal{N}, -a_{max}, a_{max})aexec​clip(μ(s)N,−amax​,amax​)本项目中使用高斯噪声 (Gaussian Noise)并随训练进行衰减。高斯噪声代码(gemini3pro生成)classGaussianNoise:def__init__(self,action_dim,sigma0.1):self.action_dimaction_dim self.sigmasigma# 标准差控制噪声大小defsample(self):# 生成标准正态分布噪声 * sigmareturnnp.random.normal(0,self.sigma,sizeself.action_dim)软更新 (Soft Update)不同于 DQN 的硬更新DDPG 采用软更新来缓慢更新目标网络参数θ ′ ← τ θ ( 1 − τ ) θ ′ \theta \leftarrow \tau \theta (1 - \tau) \thetaθ′←τθ(1−τ)θ′其中τ \tauτ通常取极小值 (如 0.005亲测选择0.01在当前环境下模型训练效果不佳)。代码实现模型定义 (Actor Critic)注意这里Critic网络输出的是Q ( s , a ) Q(s, a)Q(s,a)因此输入层节点个数为状态与动作维度之和fromtorchimportnnimporttorchimporttorch.nn.functionalasFclassActor(nn.Module):def__init__(self,state_dim,action_dim,max_action,hidden_dim256):super(Actor,self).__init__()self.netnn.Sequential(nn.Linear(state_dim,hidden_dim),nn.ReLU(),nn.Linear(hidden_dim,hidden_dim),nn.ReLU(),nn.Linear(hidden_dim,action_dim),nn.Tanh(),)self.max_actionmax_actiondefforward(self,state):returnself.net(state)*self.max_actiondefact(self,state):returnself.forward(state)classCritic(nn.Module):def__init__(self,state_dim,action_dim,hidden_dim256):super(Critic,self).__init__()self.netnn.Sequential(nn.Linear(state_dimaction_dim,hidden_dim),nn.ReLU(),nn.Linear(hidden_dim,hidden_dim),nn.ReLU(),nn.Linear(hidden_dim,1),)defforward(self,state,action):returnself.net(torch.cat([state,action],dim1))DDPG 类完整代码classDDPG:def__init__(self,state_dim,action_dim,max_action,devicecudaiftorch.cuda.is_available()elsecpu,hidden_dim256,batch_size256,gamma0.99,tau0.001,replay_buffer_size5000,actor_lr3e-4,critic_lr3e-4):self.devicedevice self.batch_sizebatch_size self.gammagamma self.tautau self.max_actionmax_action self.replay_bufferReplayBuffer(state_dim,action_dim,replay_buffer_size)self.actorActor(state_dim,action_dim,max_action,hidden_dim).to(self.device)self.target_actorActor(state_dim,action_dim,max_action,hidden_dim).to(self.device)self.target_actor.load_state_dict(self.actor.state_dict())self.criticCritic(state_dim,action_dim,hidden_dim).to(self.device)self.target_criticCritic(state_dim,action_dim,hidden_dim).to(self.device)self.target_critic.load_state_dict(self.critic.state_dict())self.actor_optimizeroptim.Adam(self.actor.parameters(),lractor_lr)self.critic_optimizeroptim.Adam(self.critic.parameters(),lrcritic_lr)defact(self,state):statetorch.FloatTensor(state).unsqueeze(0).to(self.device)withtorch.no_grad():actionself.actor(state)returnaction.cpu().numpy().flatten()defstore_transition(self,state,action,reward,next_state,done):self.replay_buffer.add(state,action,reward,next_state,done)defsample(self):returnself.replay_buffer.sample(self.batch_size)deftrain(self):ifself.replay_buffer.sizeself.batch_size:returnstates,actions,rewards,next_states,donesself.replay_buffer.sample(self.batch_size,deviceself.device)withtorch.no_grad():next_actionsself.target_actor(next_states)td_targetsrewardsself.gamma*self.target_critic(next_states,next_actions)*(1-dones)current_qself.critic(states,actions)critic_lossF.mse_loss(current_q,td_targets)actor_loss-self.critic(states,self.actor(states)).mean()self.actor_optimizer.zero_grad()actor_loss.backward()self.actor_optimizer.step()self.critic_optimizer.zero_grad()critic_loss.backward()self.critic_optimizer.step()forparam,target_paraminzip(self.actor.parameters(),self.target_actor.parameters()):target_param.data.copy_(param.data*self.tautarget_param.data*(1-self.tau))forparam,target_paraminzip(self.critic.parameters(),self.target_critic.parameters()):target_param.data.copy_(param.data*self.tautarget_param.data*(1-self.tau))defsave(self,filename): 保存所有网络参数和优化器状态到一个文件 torch.save({actor:self.actor.state_dict(),critic:self.critic.state_dict(),target_actor:self.target_actor.state_dict(),target_critic:self.target_critic.state_dict(),actor_optimizer:self.actor_optimizer.state_dict(),critic_optimizer:self.critic_optimizer.state_dict(),},filename)defload(self,filename): 加载模型参数 # map_location 确保在 CPU 机器上也能加载 GPU 训练的模型反之亦然checkpointtorch.load(filename,map_locationself.device)self.actor.load_state_dict(checkpoint[actor])self.critic.load_state_dict(checkpoint[critic])self.target_actor.load_state_dict(checkpoint[target_actor])self.target_critic.load_state_dict(checkpoint[target_critic])self.actor_optimizer.load_state_dict(checkpoint[actor_optimizer])self.critic_optimizer.load_state_dict(checkpoint[critic_optimizer])训练流程训练中加入了高斯噪声衰减机制和预热Warmup阶段以平衡探索与利用。预热设置在前5000步内使用系统的随机动作并且不进行模型训练。importgymnasiumasgym,torch,numpyasnp,matplotlib.pyplotaspltfromALG.DRL.DDPGimportDDPGfromtqdmimporttqdmfromUtils.NoiseimportGaussianNoisefromUtils.SmoothimportSmooth envgym.make(LunarLander-v3,continuousTrue,render_modeNone)state_dimenv.observation_space.shape[0]iflen(env.observation_space.shape)1elseenv.observation_space.n action_dimenv.action_space.shape[0]max_actionfloat(env.action_space.high[0])modelDDPG(state_dim,action_dim,max_action,replay_buffer_size100000,tau0.005,actor_lr1e-4,batch_size512)noiseGaussianNoise(action_dim)scores[]episodes3000step0warmup_steps5000noise_decay0.999min_noise0.01max_value200update_interval4pbartqdm(range(episodes),descTraining)forepisodeinpbar:ifstepwarmup_steps:noise.sigmamax(min_noise,noise.sigma*noise_decay)doneFalsestate,_env.reset()score0whilenotdone:step1ifstepwarmup_steps:actionenv.action_space.sample()else:actionmodel.act(state)action(actionnoise.sample()).clip(-max_action,max_action)next_state,reward,termination,truncated,_env.step(action)doneterminationortruncated scorereward model.store_transition(state,action,reward,next_state,done)statenext_stateifstepwarmup_stepsandstep%update_interval0:model.train()scores.append(score)pbar.set_postfix(epepisode,scoref{score:.2f},avg100f{np.mean(scores[-100:]):.2f})ifnp.mean(scores[-100:])max_value:model.save(../../model/lunarLanderContinuous-DDPG.pth)smoothSmooth(scores)smooth.show(titleDDPG in LunarLander-v3-continuous)训练结果经过参数调整增大 Batch Size 至 512增大 Buffer 至 100000模型成功收敛。可以看到模型在前期约前800轮处于探索阶段分数较低在预热结束且 Buffer 充足后分数迅速上升最终稳定在 200 分以上实现了平稳着陆。