基于多智能体迁移强化学习算法的电力系统最优碳–能复合流求解
|
摘要
为避免碳排放责任的重复计算,首次在电力系统最优碳–能复合流模型中提出发电侧、电网侧、用户侧之间的碳排放责任分摊机制。并进一步提出一种全新的多智能体迁移强化学习算法,以实现电力系统最优碳–能复合流模型的快速、高质量求解。此算法同时组织多个智能体执行优化任务,并将知识学习机制、多智能体交互机制和知识迁移机制相结合,不仅使每个智能体都具有较强的自主学习能力,还通过多个智能体之间的协调实现了问题的合作求解;知识迁移可以复用历史任务学习经验,使新任务学习效率大幅提升。IEEE 57节点系统、IEEE 300节点系统及深圳电网模型仿真结果均表明,此算法在保证最优解质量和寻优稳定性的同时,收敛速度可达其他算法的4.7～50.5倍,具有明显的优势和实用价值。
An apportionment responsibility of carbon emission among the generation side, the power grid side and the demand side is firstly considered in an optimal carbon-energy combined flow(OCECF) model, so that a double counting of carbon emission responsibility can be eliminated. Furthermore, a novel multi-agent transfer reinforcement learning(MATRL) is proposed to obtain a better solution of OCECF quickly. The proposed algorithm organizes multiple agents to perform optimization tasks, and combines knowledge learning mechanism, multi-agent interaction mechanism and knowledge transfer mechanism together. The proposed algorithm not only makes each agent have a strong self-learning ability, but also a cooperative solution of problem can be achieved through the coordination of multiple agents. Besides,the history learning experience can be reused by knowledge transfer, so that the convergence of the new task can be dramatically accelerated. The performance of MATRL has been evaluated for OCECF on IEEE 57-bus system, IEEE 300-bus system, and Shenzhen power grid model, respectively. The simulation results demonstrate that MATRL achieves a desirable performance on solution quality and optimization stability, which converges 4.7 to 50.5 times faster than conventional artificial intelligence algorithms, showing obvious advantages and practical value.
An apportionment responsibility of carbon emission among the generation side, the power grid side and the demand side is firstly considered in an optimal carbon-energy combined flow(OCECF) model, so that a double counting of carbon emission responsibility can be eliminated. Furthermore, a novel multi-agent transfer reinforcement learning(MATRL) is proposed to obtain a better solution of OCECF quickly. The proposed algorithm organizes multiple agents to perform optimization tasks, and combines knowledge learning mechanism, multi-agent interaction mechanism and knowledge transfer mechanism together. The proposed algorithm not only makes each agent have a strong self-learning ability, but also a cooperative solution of problem can be achieved through the coordination of multiple agents. Besides,the history learning experience can be reused by knowledge transfer, so that the convergence of the new task can be dramatically accelerated. The performance of MATRL has been evaluated for OCECF on IEEE 57-bus system, IEEE 300-bus system, and Shenzhen power grid model, respectively. The simulation results demonstrate that MATRL achieves a desirable performance on solution quality and optimization stability, which converges 4.7 to 50.5 times faster than conventional artificial intelligence algorithms, showing obvious advantages and practical value.
引文
[1]程耀华,张宁,康重庆,等.低碳多能源系统的研究框架及展望[J].中国电机工程学报,2017,37(14):4060-4069.CHENG Yaohua, ZHANG Ning, KANG Chongqing, et al. Researchframeworkandprospectsoflow-carbonmultipleenergysystems[J].Proceedings of the CSEE, 2017, 37(14):4060-4069.
[2]张晓花,谢俊,赵晋泉,等.考虑风电和电动汽车等不确定性负荷的电力系统节能减排调度[J].高电压技术,2015,41(7):2408-2414.ZHANGXiaohua,XIEJun,ZHAOJinquan,etal.Energy-savingemission-reduction dispatching of electrical power system consideringuncertainty of load with wind power and plug-in hybrid electric vehi-cles[J]. High Voltage Engineering, 2015, 41(7):2408-2414.
[3]黄忠源,李进,安洪光,等.燃烧后CO2捕获与燃气–蒸汽联合循环机组热力能源整合研究[J].中国电机工程学报,2017,37(9):2644-2651.HUANG Zhongyuan, LI Jin, AN Hongguang, et al. Study on the ther-mal energy integration of post-combustion CO2 capture in natural gascombinedcycleplant[J].ProceedingsoftheCSEE,2017,37(9):2644-2651.
[4]周天睿,康重庆,徐乾耀,等.电力系统碳排放流分析理论初探[J].电力系统自动化,2012,36(7):38-43.ZHOUTianrui,KANGChongqing,XUQianyao,etal.Preliminarytheoreticalinvestigationonpowersystemcarbonemissionflow[J].Automation of Electric Power Systems, 2012, 36(7):38-43.
[5]周天睿,康重庆,徐乾耀,等.电力系统碳排放流的计算方法初探[J].电力系统自动化,2012,36(11):44-49.ZHOUTianrui,KANGChongqing,XUQianyao,etal.Preliminaryinvestigationonamethodforcarbonemissionflowcalculationofpower system[J]. Automation of Electric Power Systems, 2012, 36(11):44-49.
[6]ZHANG X, YU T, YANG B. Approximate ideal multi-objective solu-tionQ(λ)learningforoptimalcarbon-energycombined-flowinmulti-energypowersystems[J].EnergyConversion&Management,2015, 106(2):543-556.
[7]LENZEN M, MURRAY J, SACK F, et al. Shared producer and con-sumerresponsibility-theoryandpractice[J].EcologicalEconomics,2007, 61(1):27-42.
[8]孙秋野,陈会敏,杨家农,等.牛顿类潮流计算方法的收敛性分析[J].中国电机工程学报,2014,34(13):2196-2200.SUN Qiuye, CHEN Huimin, YANG Jianong, et al. Analysis on con-vergence of Newton-like power flow algorithm[J]. Proceedings of theCSEE, 2014, 34(13):2196-2200.
[9]吴国祥,孙继国,顾菊平.大规模海上风电场多端直流输电系统的最优电压控制[J].高电压技术,2016,42(9):2733-2739.WU Guoxiang, SUN Jiguo, GU Juping. Optimum voltage control formulti-terminalHVDCtransmissionsystemsconnectedtolargeoff-shorewindfarms[J].HighVoltageEngineering,2016,42(9):2733-2739.
[10]冯凯,应展烽,张旭东,等.基于内点法参数辨识的架空导线径向热路模型[J].高电压技术,2015,41(7):2321-2326.FENG Kai, YING Zhanfeng, ZHANG Xudong, et al. Radial thermalcircuit model of overhead conductors based on parameter identifica-tionwithinteriorpointmethod[J].HighVoltageEngineering,2015,41(7):2321-2326.
[11]阮羚,徐碧川,全江涛,等.用于土壤分层电阻率模型反演的人工蜂群结合混沌搜索算子及混沌算法[J].高电压技术,2015,41(1):42-48.RUAN Ling, XU Bichuan, QUAN Jiangtao, et al. Artificial colony al-gorithm combined with chaotic search operator and chaotic pool usedfor resistivity model inversion of layered soil[J]. High Voltage Engi-neering, 2015, 41(1):42-48.
[12]HE S, WU Q H, SAUNDERS J R. Group search optimizer:an opti-mizationalgorithminspiredbyanimalsearchingbehavior[J].IEEETransactions on Evolutionary Computation, 2009, 13(5):973-990.
[13]孙鹏,曹雨晨,刘洋,等.采用二进制蚁群模糊神经网络的配电网故障分类方法[J].高电压技术,2016,42(7):2063-2072.SUN Peng, CAO Yuchen, LIU Yang, et al. Fault classification tech-niqueforpowerdistributionnetworkusingbinaryantcolonyalgorithmandfuzzyneuralnetwork[J].HighVoltageEngineering,2016, 42(7):2063-2072.
[14]金立军,薛飞,彭陈仡,等.基于粒子群算法的变电站工频电场]优化[J].高电压技术,2017,43(4):1341-1347.JIN Lijun, XUE Fei, PENG Chenyi, et al. Optimization for substationpower frequency electric field distribution based on particle swarm op-timization[J]. High Voltage Engineering, 2017, 43(4):1341-1347.
[15]黄新波,李文君子,宋桐,等.采用遗传算法优化装袋分类回归树组合算法的变压器故障诊断[J].高电压技术,2016,42(5):1617-1623.HUANG Xinbo, LI Wenjunzi, SONG Tong, et al. Application of bag-ging-cartalgorithmoptimizedbygeneticalgorithmintransformerfault diagnosis[J]. High Voltage Engineering, 2016, 42(5):1617-1623.
[16]WATKINSCJCH,DAYANP.Q-learning[J].MachineLearning,1992, 8(3/4):279-292.
[17]王怀智,余涛,唐捷.基于多智能体相关均衡算法的自动发电控制[J].中国电机工程学报,2014,34(4):620-627.WANG Huaizhi, YU Tao, TANG Jie. Automatic generation control forinterconnected power grids based on multi-agent correlated equilibri-umlearningsystem[J].ProceedingsoftheCSEE,2014,34(4):620-627.
[18]YU T, ZHOU B, CHAN K W, et al. R(λ)imitation learning for auto-matic generation control of interconnected power grids[J]. Automatica,2012, 48(9):2130-2136.
[19]范波,张雷.多智能体机器人系统信息融合与协调[M].北京:科学出版社,2015:60-61.FAN Bo, ZHANG Lei. Information fusion and coordination of mul-ti-ageit robot system[M]. Beijing, China:Science Press, 2015:60-61.
[20]PAN S J, YANG Q. A survey on transfer learning[J]. IEEE Transac-tions on Knowledge&Data Engineering, 2010, 22(10):1345-1359.
[21]TAYLOR M E, STONE P. Transfer learning for reinforcement learningdomains:asurvey[J].JournalofMachineLearningResearch,2009,10(10):1633-1685.
[22]LENZEN M. A consistent input–output formulation of shared producerandconsumerresponsibility[J].EconomicSystemsResearch,2005,17(4):365-391.
[23]李保卫,胡泽春,宋永华,等.用户侧电力碳排放强度的评估原则与模型[J].电网技术,2012,36(8):6-11.LI Baowei, HU Zechun, SONG Yonghua, et al. Principle and modelforassessmentoncarbonemissionintensitycausedbyelectricityatconsumer side[J]. Power System Technology, 2012, 36(8):6-11.
[24]碳排放权交易网.碳指标-碳配额频道[EB/OL].[2018-07-26].http://www.tanpaifang.com/tanzhibiao/201807/2662182.html.CarbonEmission TradingNetwork.Carbonindex-carbon quota sec-tion[EB/OL].[2018-07-26].http://www.tanpaifang.com/tanzhibiao/201807/2662182.html.
[25]李莉,谭忠富,李宁,等.几种发电排污权初始分配模式的对比研究[J].华东电力,2010,38(11):1789-1793.LI Li, TAN Zhongfu, LI Ning, et al. Comparative study on several dis-tributionmodelsofpowergenerationemissionright[J].EastChinaElectric Power, 2010, 38(11):1789-1793.
[26]梅天华,汤优敏,甘德强.考虑历史排放赤字的电力初始碳排放权公理化公平分配方法[J].电力系统自动化,2016,39(3):52-58.MEI Tianhua, TANG Youmin, GAN Deqiang. Axiomatic equity allo-cationmethodofinitialemissionallowanceinpowersystemconsideringhistoricalemissiondeficits[J].AutomationofElectricPower System, 2016, 39(3):52-58.
[27]WANG T, WANG X, GONG Y, et al. Initial allocation of carbon emis-sion permits in power systems[J]. Journal of Modern Power Systems&Clean Energy, 2017, 5(2):1-9.
[28]余涛,张孝顺.一种具有记忆自学习能力的快速动态寻优算法及其无功优化求解[J].中国科学:技术科学,2016,46(3):256-267.YU Tao, ZHANG Xiaoshun. A fast dynamic optimization algorithmwith memory and self-learning and its application on reactive poweroptimization[J].ScienceChinaTechnologicalSciences,2016,46(3):256-267.
[29]TOKICM.Adaptiveε-greedyexplorationinreinforcementlearningbased on value differences[C]∥German Conference onAdvances inArtificialIntelligence.Berlin,German:Springer-Verlag,2010:203-210.
[30]GAMBARDELLA L M, DORIGO M. Ant-Q:a reinforcement learningapproachtothetravelingsalesmanproblem[J].MachineLearningProceedings, 1995, 170(3):252-260.
[31]ATASHPAZ-GARGARIE,LUCASC.Imperialistcompetitivealgo-rithm:analgorithmforoptimizationinspiredbyimperialisticcompetition[C]∥IEEECongressonEvolutionaryComputation.Sin-gapore:IEEE, 2007:4661-4667.
[32]COELLO C A C. Theoretical and numerical constraint-handling tech-niques used with evolutionary algorithms:a survey of the state of theart[J]. Computer Methods in Applied Mechanics&Engineering, 2002,191(11):1245-1287.
[33]MIRJALILIS.Theantlionoptimizer[J].AdvancesinEngineeringSoftware, 2015, 83(5):80-98.
[34]RAO R V, PATEL V. An elitist teaching-learning-based optimizationalgorithm for solving complex constrained optimization problems[J].InternationalJournalofIndustrialEngineeringComputations,2012,3(4):535-560.
[35]徐方维,谭洋洋,杨洪耕,等.兼顾不同角色利益的集中型充电站优化布局[J].高电压技术,2017,43(4):1256-1262.XU Fangwei, TAN Yangyang, YANG Honggeng, et al. Optimal plan-ning of centralized charging stations considering benefits for differentroles[J]. High Voltage Engineering, 2017, 43(4):1256-1262.
[36]陈光.节能减排环境下的电力系统规划与重构相关问题研究[D].杭州:浙江大学,2013.CHEN Guang. Study on relevant issues of power system planning andreconfiguration in the environment of energy saving and emission re-duction[D]. Hangzhou, China:Zhejiang University, 2013.
[37]FANGKT,MAC,WINKERP,etal.Uniformdesign:theoryandapplication[J]. Technometrics, 2000, 42(3):237-248.
[2]张晓花,谢俊,赵晋泉,等.考虑风电和电动汽车等不确定性负荷的电力系统节能减排调度[J].高电压技术,2015,41(7):2408-2414.ZHANGXiaohua,XIEJun,ZHAOJinquan,etal.Energy-savingemission-reduction dispatching of electrical power system consideringuncertainty of load with wind power and plug-in hybrid electric vehi-cles[J]. High Voltage Engineering, 2015, 41(7):2408-2414.
[3]黄忠源,李进,安洪光,等.燃烧后CO2捕获与燃气–蒸汽联合循环机组热力能源整合研究[J].中国电机工程学报,2017,37(9):2644-2651.HUANG Zhongyuan, LI Jin, AN Hongguang, et al. Study on the ther-mal energy integration of post-combustion CO2 capture in natural gascombinedcycleplant[J].ProceedingsoftheCSEE,2017,37(9):2644-2651.
[4]周天睿,康重庆,徐乾耀,等.电力系统碳排放流分析理论初探[J].电力系统自动化,2012,36(7):38-43.ZHOUTianrui,KANGChongqing,XUQianyao,etal.Preliminarytheoreticalinvestigationonpowersystemcarbonemissionflow[J].Automation of Electric Power Systems, 2012, 36(7):38-43.
[5]周天睿,康重庆,徐乾耀,等.电力系统碳排放流的计算方法初探[J].电力系统自动化,2012,36(11):44-49.ZHOUTianrui,KANGChongqing,XUQianyao,etal.Preliminaryinvestigationonamethodforcarbonemissionflowcalculationofpower system[J]. Automation of Electric Power Systems, 2012, 36(11):44-49.
[6]ZHANG X, YU T, YANG B. Approximate ideal multi-objective solu-tionQ(λ)learningforoptimalcarbon-energycombined-flowinmulti-energypowersystems[J].EnergyConversion&Management,2015, 106(2):543-556.
[7]LENZEN M, MURRAY J, SACK F, et al. Shared producer and con-sumerresponsibility-theoryandpractice[J].EcologicalEconomics,2007, 61(1):27-42.
[8]孙秋野,陈会敏,杨家农,等.牛顿类潮流计算方法的收敛性分析[J].中国电机工程学报,2014,34(13):2196-2200.SUN Qiuye, CHEN Huimin, YANG Jianong, et al. Analysis on con-vergence of Newton-like power flow algorithm[J]. Proceedings of theCSEE, 2014, 34(13):2196-2200.
[9]吴国祥,孙继国,顾菊平.大规模海上风电场多端直流输电系统的最优电压控制[J].高电压技术,2016,42(9):2733-2739.WU Guoxiang, SUN Jiguo, GU Juping. Optimum voltage control formulti-terminalHVDCtransmissionsystemsconnectedtolargeoff-shorewindfarms[J].HighVoltageEngineering,2016,42(9):2733-2739.
[10]冯凯,应展烽,张旭东,等.基于内点法参数辨识的架空导线径向热路模型[J].高电压技术,2015,41(7):2321-2326.FENG Kai, YING Zhanfeng, ZHANG Xudong, et al. Radial thermalcircuit model of overhead conductors based on parameter identifica-tionwithinteriorpointmethod[J].HighVoltageEngineering,2015,41(7):2321-2326.
[11]阮羚,徐碧川,全江涛,等.用于土壤分层电阻率模型反演的人工蜂群结合混沌搜索算子及混沌算法[J].高电压技术,2015,41(1):42-48.RUAN Ling, XU Bichuan, QUAN Jiangtao, et al. Artificial colony al-gorithm combined with chaotic search operator and chaotic pool usedfor resistivity model inversion of layered soil[J]. High Voltage Engi-neering, 2015, 41(1):42-48.
[12]HE S, WU Q H, SAUNDERS J R. Group search optimizer:an opti-mizationalgorithminspiredbyanimalsearchingbehavior[J].IEEETransactions on Evolutionary Computation, 2009, 13(5):973-990.
[13]孙鹏,曹雨晨,刘洋,等.采用二进制蚁群模糊神经网络的配电网故障分类方法[J].高电压技术,2016,42(7):2063-2072.SUN Peng, CAO Yuchen, LIU Yang, et al. Fault classification tech-niqueforpowerdistributionnetworkusingbinaryantcolonyalgorithmandfuzzyneuralnetwork[J].HighVoltageEngineering,2016, 42(7):2063-2072.
[14]金立军,薛飞,彭陈仡,等.基于粒子群算法的变电站工频电场]优化[J].高电压技术,2017,43(4):1341-1347.JIN Lijun, XUE Fei, PENG Chenyi, et al. Optimization for substationpower frequency electric field distribution based on particle swarm op-timization[J]. High Voltage Engineering, 2017, 43(4):1341-1347.
[15]黄新波,李文君子,宋桐,等.采用遗传算法优化装袋分类回归树组合算法的变压器故障诊断[J].高电压技术,2016,42(5):1617-1623.HUANG Xinbo, LI Wenjunzi, SONG Tong, et al. Application of bag-ging-cartalgorithmoptimizedbygeneticalgorithmintransformerfault diagnosis[J]. High Voltage Engineering, 2016, 42(5):1617-1623.
[16]WATKINSCJCH,DAYANP.Q-learning[J].MachineLearning,1992, 8(3/4):279-292.
[17]王怀智,余涛,唐捷.基于多智能体相关均衡算法的自动发电控制[J].中国电机工程学报,2014,34(4):620-627.WANG Huaizhi, YU Tao, TANG Jie. Automatic generation control forinterconnected power grids based on multi-agent correlated equilibri-umlearningsystem[J].ProceedingsoftheCSEE,2014,34(4):620-627.
[18]YU T, ZHOU B, CHAN K W, et al. R(λ)imitation learning for auto-matic generation control of interconnected power grids[J]. Automatica,2012, 48(9):2130-2136.
[19]范波,张雷.多智能体机器人系统信息融合与协调[M].北京:科学出版社,2015:60-61.FAN Bo, ZHANG Lei. Information fusion and coordination of mul-ti-ageit robot system[M]. Beijing, China:Science Press, 2015:60-61.
[20]PAN S J, YANG Q. A survey on transfer learning[J]. IEEE Transac-tions on Knowledge&Data Engineering, 2010, 22(10):1345-1359.
[21]TAYLOR M E, STONE P. Transfer learning for reinforcement learningdomains:asurvey[J].JournalofMachineLearningResearch,2009,10(10):1633-1685.
[22]LENZEN M. A consistent input–output formulation of shared producerandconsumerresponsibility[J].EconomicSystemsResearch,2005,17(4):365-391.
[23]李保卫,胡泽春,宋永华,等.用户侧电力碳排放强度的评估原则与模型[J].电网技术,2012,36(8):6-11.LI Baowei, HU Zechun, SONG Yonghua, et al. Principle and modelforassessmentoncarbonemissionintensitycausedbyelectricityatconsumer side[J]. Power System Technology, 2012, 36(8):6-11.
[24]碳排放权交易网.碳指标-碳配额频道[EB/OL].[2018-07-26].http://www.tanpaifang.com/tanzhibiao/201807/2662182.html.CarbonEmission TradingNetwork.Carbonindex-carbon quota sec-tion[EB/OL].[2018-07-26].http://www.tanpaifang.com/tanzhibiao/201807/2662182.html.
[25]李莉,谭忠富,李宁,等.几种发电排污权初始分配模式的对比研究[J].华东电力,2010,38(11):1789-1793.LI Li, TAN Zhongfu, LI Ning, et al. Comparative study on several dis-tributionmodelsofpowergenerationemissionright[J].EastChinaElectric Power, 2010, 38(11):1789-1793.
[26]梅天华,汤优敏,甘德强.考虑历史排放赤字的电力初始碳排放权公理化公平分配方法[J].电力系统自动化,2016,39(3):52-58.MEI Tianhua, TANG Youmin, GAN Deqiang. Axiomatic equity allo-cationmethodofinitialemissionallowanceinpowersystemconsideringhistoricalemissiondeficits[J].AutomationofElectricPower System, 2016, 39(3):52-58.
[27]WANG T, WANG X, GONG Y, et al. Initial allocation of carbon emis-sion permits in power systems[J]. Journal of Modern Power Systems&Clean Energy, 2017, 5(2):1-9.
[28]余涛,张孝顺.一种具有记忆自学习能力的快速动态寻优算法及其无功优化求解[J].中国科学:技术科学,2016,46(3):256-267.YU Tao, ZHANG Xiaoshun. A fast dynamic optimization algorithmwith memory and self-learning and its application on reactive poweroptimization[J].ScienceChinaTechnologicalSciences,2016,46(3):256-267.
[29]TOKICM.Adaptiveε-greedyexplorationinreinforcementlearningbased on value differences[C]∥German Conference onAdvances inArtificialIntelligence.Berlin,German:Springer-Verlag,2010:203-210.
[30]GAMBARDELLA L M, DORIGO M. Ant-Q:a reinforcement learningapproachtothetravelingsalesmanproblem[J].MachineLearningProceedings, 1995, 170(3):252-260.
[31]ATASHPAZ-GARGARIE,LUCASC.Imperialistcompetitivealgo-rithm:analgorithmforoptimizationinspiredbyimperialisticcompetition[C]∥IEEECongressonEvolutionaryComputation.Sin-gapore:IEEE, 2007:4661-4667.
[32]COELLO C A C. Theoretical and numerical constraint-handling tech-niques used with evolutionary algorithms:a survey of the state of theart[J]. Computer Methods in Applied Mechanics&Engineering, 2002,191(11):1245-1287.
[33]MIRJALILIS.Theantlionoptimizer[J].AdvancesinEngineeringSoftware, 2015, 83(5):80-98.
[34]RAO R V, PATEL V. An elitist teaching-learning-based optimizationalgorithm for solving complex constrained optimization problems[J].InternationalJournalofIndustrialEngineeringComputations,2012,3(4):535-560.
[35]徐方维,谭洋洋,杨洪耕,等.兼顾不同角色利益的集中型充电站优化布局[J].高电压技术,2017,43(4):1256-1262.XU Fangwei, TAN Yangyang, YANG Honggeng, et al. Optimal plan-ning of centralized charging stations considering benefits for differentroles[J]. High Voltage Engineering, 2017, 43(4):1256-1262.
[36]陈光.节能减排环境下的电力系统规划与重构相关问题研究[D].杭州:浙江大学,2013.CHEN Guang. Study on relevant issues of power system planning andreconfiguration in the environment of energy saving and emission re-duction[D]. Hangzhou, China:Zhejiang University, 2013.
[37]FANGKT,MAC,WINKERP,etal.Uniformdesign:theoryandapplication[J]. Technometrics, 2000, 42(3):237-248.