Single node clusters are Dataproc clusters with only one node. This singlenode acts as the master and worker for your Dataproc cluster. While singlenode clusters only have one node, most Dataproc concepts and featuresstill apply, except those listed below.
There are a number of situations where single node Dataproc clusters canbe useful, including:
- Trying out new versions of Spark and Hadoop or other open source components
- Building proof-of-concept (PoC) demonstrations
- Lightweight data science
- Small-scale non-critical data processing
- Education related to the Spark and Hadoop ecosystem
Single node cluster semantics
The following semantics apply to single node Dataproc clusters:
- Single node clusters are configured the same as multi node Dataprocclusters, and include services such as HDFS and YARN.
- Single node clusters report as master nodes forinitialization actions.
- Single node clusters show 0 workers since the single node acts asboth master and worker.
- Single node clusters are given hostnames that follow the pattern
clustername-m.You can use this hostname to SSH into or connect to aweb UI on the node. - Single node clusters cannot be upgraded to multi node clusters. Once created,single node clusters are restricted to one node. Similarly, multi nodeclusters cannot be scaled down to single node clusters.
Limitations
Single node clusters are not recommended for large-scale parallel dataprocessing. If you exceed the resources on a single node cluster, a multi nodeDataproc cluster is recommended.
Single node clusters are not available withhigh-availabilitysince there is only one node in the cluster.
Single node clusters cannot use preemptible VMs.
Creating a single node cluster
gcloud command
You can create a single node Dataproc cluster using the gcloudcommand-line tool. To create a single node cluster, pass the--single-node flag to thegcloud dataproc clusters createcommand.
gcloud dataproc clusters create cluster-name \--region=region \--single-node \... other args
REST API
You can create a single node cluster through theDataproc REST API using aclusters.createrequest. When making this request, you must:
- Add the property
"dataproc:dataproc.allow.zero.workers":"true"to theSoftwareConfigof the cluster request. - Don't submit values for
workerConfigandsecondaryWorkerConfig(see ClusterConfig).
Console
You can create a single node cluster by selecting "Single Node(1 master, 0 workers)" on the Cluster type section ofthe Set up cluster panel on the DataprocCreate a clusterpage.
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2024-09-10 UTC.
[{ "type": "thumb-down", "id": "hardToUnderstand", "label":"Hard to understand" },{ "type": "thumb-down", "id": "incorrectInformationOrSampleCode", "label":"Incorrect information or sample code" },{ "type": "thumb-down", "id": "missingTheInformationSamplesINeed", "label":"Missing the information/samples I need" },{ "type": "thumb-down", "id": "otherDown", "label":"Other" }] [{ "type": "thumb-up", "id": "easyToUnderstand", "label":"Easy to understand" },{ "type": "thumb-up", "id": "solvedMyProblem", "label":"Solved my problem" },{ "type": "thumb-up", "id": "otherUp", "label":"Other" }]
