Allocating MapReduce workflows with deadlines to heterogeneous servers in a cloud data center

Vista sencilla de ítem
dc.contributor.authorWang, Jiaes_ES
dc.contributor.authorLi, Xiaopinges_ES
dc.contributor.authorRuiz García, Rubénes_ES
dc.contributor.authorXu, Hanchuanes_ES
dc.contributor.authorChu, Dianhuies_ES
dc.contributor.funderAGENCIA ESTATAL DE INVESTIGACIONes_ES
dc.contributor.funderEuropean Regional Development Fundes_ES
dc.contributor.funderNational Natural Science Foundation of Chinaes_ES
dc.contributor.funderNational Key Research and Development Program of Chinaes_ES
dc.date.accessioned2021-11-05T14:06:42Z
dc.date.available2021-11-05T14:06:42Z
dc.date.issued2020-06es_ES
dc.description.abstract[EN] Total profit is one of the most important factors to be considered from the perspective of resource providers. In this paper, an original MapReduce workflow scheduling with deadline and data locality is proposed to maximize total profit of resource providers. A new workflow conversion based on dynamic programming and ChainMap/ChainReduce is designed to decrease transmission times among MapReduce jobs of workflows. A new deadline division considering execution time, float time and job level is proposed to obtain better deadlines of MapReduce jobs in workflows. With the adapted replica strategy in MapReduce workflow, a new task scheduling is proposed to improve data locality which assigns tasks to servers with the earliest completion time in order to ensure resource providers obtain more profit. Experimental results show that the proposed heuristic results in larger total profit than other adopted algorithms.en_EN
dc.description.accrualMethodSes_ES
dc.description.bibliographicCitationWang, J.; Li, X.; Ruiz García, R.; Xu, H.; Chu, D. (2020). Allocating MapReduce workflows with deadlines to heterogeneous servers in a cloud data center. Service Oriented Computing and Applications. 14(2):101-118. https://doi.org/10.1007/s11761-020-00290-1es_ES
dc.description.issue2es_ES
dc.description.referencesZaharia M, Chowdhury M, Franklin M et al (2010) Spark: cluster computing with working sets. In: Usenix conference on hot topics in cloud computing, pp 1765–1773es_ES
dc.description.referencesLi L, Ma Z, Liu L et al (2013) Hadoop-based ARIMA algorithm and its application in weather forecast. Int J Database Theory Appl 6(5):119–132es_ES
dc.description.referencesXun Y, Zhang J, Qin X (2017) FiDoop: parallel mining of frequent itemsets using MapReduce. IEEE Trans Syst Man Cybern Syst 46(3):313–325es_ES
dc.description.referencesWang Y, Shi W (2014) Budget-driven scheduling algorithms for batches of MapReduce jobs in heterogeneous clouds. IEEE Trans Cloud Comput 2(3):306–319es_ES
dc.description.referencesTiwari N, Sarkar S, Bellur U et al (2015) Classification framework of MapReduce scheduling algorithms. ACM Comput Surv 47(3):1–49es_ES
dc.description.referencesBu Y, Howe B, Balazinska M et al (2012) The HaLoop approach to large-scale iterative data analysis. VLDB J 21(2):169–190es_ES
dc.description.referencesGunarathne T, Zhang B, Wu T et al (2013) Scalable parallel computing on clouds using Twister4Azure iterative MapReduce. Future Gener Comput Syst 29(4):1035–1048es_ES
dc.description.referencesZhang Y, Gao Q, Gao L et al (2012) iMapReduce: a distributed computing framework for iterative computation. J Grid Comput 10(1):47–68es_ES
dc.description.referencesDong X, Wang Y, Liao H (2011) Scheduling mixed real-time and non-real-time applications in MapReduce environment. In: International conference on parallel and distributed systems, pp 9–16es_ES
dc.description.referencesTang Z, Zhou J, Li K et al (2013) A MapReduce task scheduling algorithm for deadline constraints. Clust Comput 16(4):651–662es_ES
dc.description.referencesZhang W, Rajasekaran S, Wood T et al (2014) MIMP: deadline and interference aware scheduling of Hadoop virtual machines. In: International symposium on cluster, cloud and grid computing, pp 394–403es_ES
dc.description.referencesTeng F, Magoulès F, Yu L et al (2014) A novel real-time scheduling algorithm and performance analysis of a MapReduce-based cloud. J Supercomput 69(2):739–765es_ES
dc.description.referencesPalanisamy B, Singh A, Liu L (2015) Cost-effective resource provisioning for MapReduce in a cloud. IEEE Trans Parallel Distrib Syst 26(5):1265–1279es_ES
dc.description.referencesHashem I, Anuar N, Marjani M et al (2018) Multi-objective scheduling of MapReduce jobs in big data processing. Multimed Tools Appl 77(8):9979–9994es_ES
dc.description.referencesXu X, Tang M, Tian Y (2017) QoS-guaranteed resource provisioning for cloud-based MapReduce in dynamical environments. Future Gener Comput Syst 78(1):18–30es_ES
dc.description.referencesLi H, Wei X, Fu Q et al (2014) MapReduce delay scheduling with deadline constraint. Concurr Comput Pract Exp 26(3):766–778es_ES
dc.description.referencesPolo J, Becerra Y, Carrera D et al (2013) Deadline-based MapReduce workload management. IEEE Trans Netw Serv Manag 10(2):231–244es_ES
dc.description.referencesChen C, Lin J, Kuo S (2018) MapReduce scheduling for deadline-constrained jobs in heterogeneous cloud computing systems. IEEE Trans Cloud Comput 6(1):127–140es_ES
dc.description.referencesKao Y, Chen Y (2016) Data-locality-aware MapReduce real-time scheduling framework. J Syst Softw 112:65–77es_ES
dc.description.referencesBok K, Hwang J, Lim J et al (2017) An efficient MapReduce scheduling scheme for processing large multimedia data. Multimed Tools Appl 76(16):1–24es_ES
dc.description.referencesChen Y, Borthakur D, Borthakur D et al (2012) Energy efficiency for large-scale MapReduce workloads with significant interactive analysis. In: ACM european conference on computer systems, pp 43–56es_ES
dc.description.referencesMashayekhy L, Nejad M, Grosu D et al (2015) Energy-aware scheduling of MapReduce jobs for big data applications. IEEE Trans Parallel Distrib Syst 26(10):2720–2733es_ES
dc.description.referencesLei H, Zhang T, Liu Y et al (2015) SGEESS: smart green energy-efficient scheduling strategy with dynamic electricity price for data center. J Syst Softw 108:23–38es_ES
dc.description.referencesOliveira D, Ocana K, Baiao F et al (2012) A provenance-based adaptive scheduling heuristic for parallel scientific workflows in clouds. J Grid Comput 10(3):521–552es_ES
dc.description.referencesLi S, Hu S, Abdelzaher T (2015) The packing server for real-time scheduling of MapReduce workflows. In: IEEE real-time and embedded technology and applications symposium, pp 51–62es_ES
dc.description.referencesCai Z, Li X, Ruiz R et al (2017) A delay-based dynamic scheduling algorithm for bag-of-task workflows with stochastic task execution times in clouds. Future Gener Comput Syst 71:57–72es_ES
dc.description.referencesCai Z, Li X, Ruiz R (2017) Resource provisioning for task-batch based workflows with deadlines in public clouds. IEEE Trans Cloud Comput. https://doi.org/10.1109/TCC.2017.2663426es_ES
dc.description.referencesCai Z, Li X, Gupta J (2016) Heuristics for provisioning services to workflows in XaaS clouds. IEEE Trans Serv Comput 9(2):250–263es_ES
dc.description.referencesLi X, Cai Z (2017) Elastic resource provisioning for cloud workflow applications. IEEE Trans Autom Sci Eng 14(2):1195–1210es_ES
dc.description.referencesTang Z, Liu M, Ammar A et al (2014) An optimized MapReduce workflow scheduling algorithm for heterogeneous computing. J Supercomput 72(6):1–21es_ES
dc.description.referencesXu C, Yang J, Yin K et al (2017) Optimal construction of virtual networks for cloud-based MapReduce workflows. Comput Netw 112:194–207es_ES
dc.description.referencesChiara S, Danilo A, Gianpaolo C et al (2013) Optimizing service selection and allocation in situational computing applications. IEEE Trans Serv Comput 6(3):414–428es_ES
dc.description.referencesBaresi L, Elisabetta D, Carlo G et al (2007) A framework for the deployment of adaptable web service compositions. Serv Oriented Comput Appl 1(1):75–91es_ES
dc.description.referencesLim H, Herodotou H, Babu S (2012) Stubby: a transformation-based optimizer for MapReduce workflows. VLDB Endow 5(11):1196–1207es_ES
dc.description.referencesKe H, Li P, Guo S et al (2016) On traffic-aware partition and aggregation in MapReduce for big data applications. IEEE Trans Parallel Distrib Syst 27(3):818–828es_ES
dc.description.referencesYu W, Wang Y, Que X et al (2015) Virtual shuffling for efficient data movement in MapReduce. IEEE Trans Comput 64(2):556–568es_ES
dc.description.referencesChowdhury M, Zaharia M, Ma J et al (2011) Managing data transfers in computer clusters with orchestra. ACM SIGCOMM Comput Commun 41(4):98–109es_ES
dc.description.referencesGuo D, Xie J, Zhou X et al (2015) Exploiting efficient and scalable shuffle transfers in future data center network. IEEE Trans Parallel Distrib Syst 26(4):997–1009es_ES
dc.description.referencesLi D, Yu Y, He W et al (2015) Willow: saving data center network energy for network-limited flows. IEEE Trans Parallel Distrib Syst 26(9):2610–2620es_ES
dc.description.referencesTan J, Meng X, Zhang L (2013) Coupling task progress for MapReduce resource-aware scheduling. In: IEEE INFOCOM, pp 1618–1626es_ES
dc.description.referencesHammoud M, Rehman M, Sakr M (2012) Center-of-gravity reduce task scheduling to lower MapReduce network traffic. In: International conference on cloud computing, pp 49–58es_ES
dc.description.referencesGuo Z, Fox G, Zhou M et al (2012) Improving resource utilization in MapReduce. In: International conference on cluster computing, pp 402–410es_ES
dc.description.referencesFischer M, Su X, Yin Y (2010) Assigning tasks for efficiency in Hadoop. In: Proceedings of the 22nd ACM symposium on parallelism in algorithms and architectures, pp 30–39es_ES
dc.description.referencesZhu Y, Jiang Y, Wu W et al (2014) Minimizing makespan and total completion time in MapReduce-like systems. In: IEEE INFOCOM, pp 2166–2174es_ES
dc.description.referencesKavulya S, Tan J, Gandhi R et al (2010) An analysis of traces from a production MapReduce cluster. In: IEEE/ACM international conference on cluster, cloud and grid computing, pp 94–103es_ES
dc.description.referencesAbrishami S, Naghibzadeh M, Epema D (2013) Deadline-constrained workflow scheduling algorithms for Infrastructure as a Service clouds. Future Gener Comput Syst 29(1):158–169es_ES
dc.description.referencesFernando B, Edmundo R (2010) Towards the scheduling of multiple workflows on computational grids. J Grid Comput 8(3):419–441es_ES
dc.description.referencesTiwari N, Sarkar S, Bellur U et al (2015) Classification framework of MapReduce scheduling algorithms. ACM Comput Surv 47(3):1–38es_ES
dc.description.referencesVerma A, Cherkasova L, Campbell R (2013) Orchestrating an ensemble of MapReduce jobs for minimizing their makespan. IEEE Trans Dependable Secur Comput 10(5):314–327es_ES
dc.description.referencesHeintz B, Chandra A, Sitaraman R et al (2017) End-to-end optimization for geo-distributed MapReduce. IEEE Trans Cloud Comput 4(3):293–306es_ES
dc.description.referencesChen L, Li X (2018) Cloud workflow scheduling with hybrid resource provisioning. J Supercomput 74(12):6529–6553es_ES
dc.description.referencesLi X, Jiang T, Ruiz R (2016) Heuristics for periodical batch job scheduling in a MapReduce computing framework. Inf Sci 326:119–133es_ES
dc.description.referencesVanhoucheabcd M, Maenhout B, Tavares L (2008) An evaluation of the adequacy of project network generators with systematically sampled networks. Eur J Oper Res 187(2):511–524es_ES
dc.description.sponsorshipThis work is supported by the National Key Research and Development Program of China (No. 2017YFB1400801), the National Natural Science Foundation of China (Nos. 61872077, 61832004) and Collaborative Innovation Center of Wireless Communications Technology. Rubén Ruiz is partly supported by the Spanish Ministry of Science, Innovation, and Universities, under the project ¿OPTEP-Port Terminal Operations Optimization¿ (No. RTI2018-094940-B-I00) financed with FEDER funds¿.es_ES
dc.description.upvformatpfin118es_ES
dc.description.upvformatpinicio101es_ES
dc.description.volume14es_ES
dc.identifier.doi10.1007/s11761-020-00290-1es_ES
dc.identifier.issn1863-2386es_ES
dc.identifier.urihttps://riunet.upv.es/handle/10251/176253
dc.languageIngléses_ES
dc.publisherSpringer-Verlages_ES
dc.relation.ispartofService Oriented Computing and Applicationses_ES
dc.relation.pasarelaS\424872es_ES
dc.relation.projectIDinfo:eu-repo/grantAgreement/AEI/Plan Estatal de Investigación Científica y Técnica y de Innovación 2017-2020/RTI2018-094940-B-I00/ES/OPTIMIZACION DE OPERACIONES EN TERMINALES PORTUARIAS/es_ES
dc.relation.projectIDinfo:eu-repo/grantAgreement/NSFC//61832004/es_ES
dc.relation.projectIDinfo:eu-repo/grantAgreement/NSFC//61872077/es_ES
dc.relation.projectIDinfo:eu-repo/grantAgreement/National Key Research and Development Program, China//2017YFB1400801/es_ES
dc.relation.publisherversionhttps://doi.org/10.1007/s11761-020-00290-1es_ES
dc.relation.references10.14257/ijdta.2013.6.5.11es_ES
dc.relation.references10.1109/TSMC.2015.2437327es_ES
dc.relation.references10.1109/TCC.2014.2316812es_ES
dc.relation.references10.1007/s00778-012-0269-7es_ES
dc.relation.references10.1016/j.future.2012.05.027es_ES
dc.relation.references10.1007/s10723-012-9204-9es_ES
dc.relation.references10.1109/ICPADS.2011.115es_ES
dc.relation.references10.1007/s10586-012-0236-5es_ES
dc.relation.references10.1109/CCGrid.2014.101es_ES
dc.relation.references10.1007/s11227-014-1115-zes_ES
dc.relation.references10.1109/TPDS.2014.2320498es_ES
dc.relation.references10.1007/s11042-017-4685-yes_ES
dc.relation.references10.1002/cpe.3050es_ES
dc.relation.references10.1109/TNSM.2012.122112.110163es_ES
dc.relation.references10.1109/TCC.2015.2474403es_ES
dc.relation.references10.1016/j.jss.2015.11.001es_ES
dc.relation.references10.1145/2168836.2168842es_ES
dc.relation.references10.1109/TPDS.2014.2358556es_ES
dc.relation.references10.1016/j.jss.2015.06.026es_ES
dc.relation.references10.1007/s10723-012-9227-2es_ES
dc.relation.references10.1109/RTAS.2015.7108416es_ES
dc.relation.references10.1016/j.future.2017.01.020es_ES
dc.relation.references10.1109/TCC.2017.2663426es_ES
dc.relation.references10.1109/TSC.2014.2361320es_ES
dc.relation.references10.1109/TASE.2015.2500574es_ES
dc.relation.references10.1016/j.comnet.2016.11.001es_ES
dc.relation.references10.1109/TSC.2012.18es_ES
dc.relation.references10.1007/s11761-007-0004-1es_ES
dc.relation.references10.14778/2350229.2350239es_ES
dc.relation.references10.1109/TPDS.2015.2419671es_ES
dc.relation.references10.1109/TC.2013.216es_ES
dc.relation.references10.1145/2043164.2018448es_ES
dc.relation.references10.1109/TPDS.2014.2316829es_ES
dc.relation.references10.1109/TPDS.2014.2350990es_ES
dc.relation.references10.1109/INFCOM.2013.6566958es_ES
dc.relation.references10.1109/CLOUD.2012.92es_ES
dc.relation.references10.1109/CLUSTER.2012.69es_ES
dc.relation.references10.1145/1810479.1810484es_ES
dc.relation.references10.1109/INFOCOM.2014.6848159es_ES
dc.relation.references10.1109/CCGRID.2010.112es_ES
dc.relation.references10.1016/j.future.2012.05.004es_ES
dc.relation.references10.1007/s10723-009-9144-1es_ES
dc.relation.references10.1145/2693315es_ES
dc.relation.references10.1109/TDSC.2013.14es_ES
dc.relation.references10.1109/TCC.2014.2355225es_ES
dc.relation.references10.1007/s11227-017-2043-5es_ES
dc.relation.references10.1016/j.ins.2015.07.040es_ES
dc.relation.references10.1016/j.ejor.2007.03.032es_ES
dc.rightsReserva de todos los derechoses_ES
dc.rights.accessRightsAbiertoes_ES
dc.subjectMapReduce workflow schedulinges_ES
dc.subjectHeterogeneous cloud centeres_ES
dc.subjectDeadlinees_ES
dc.subjectData locality,Profites_ES
dc.subject.classificationESTADISTICA E INVESTIGACION OPERATIVAes_ES
dc.titleAllocating MapReduce workflows with deadlines to heterogeneous servers in a cloud data centeres_ES
dc.typeArtículoes_ES
dc.type.versioninfo:eu-repo/semantics/publishedVersiones_ES
dspace.entity.typePublication
upv.uuid7fcf3703-c9ca-4b7d-82cf-90270f6b692des_ES

Archivos

Bloque original

Mostrando 1 - 2 de 2
Miniatura
Nombre:
WangLiRuiz - Allocating MapReduce workflows with deadlines to heterogeneous servers in a cloud da....pdf
Tamaño:
900.32 KB
Formato:
Adobe Portable Document Format
Descripción:
Versión del Autor.
[object Object] Descargar
Miniatura
Nombre:
Published.pdf
Tamaño:
1.4 MB
Formato:
Adobe Portable Document Format
Descripción:
Versión editorial
Descargar

Colecciones

Artículos, conferencias, monografías
OpenAIRE (Open Access Infrastructure for Research in Europe)