concept production cluster in category data

appears as: production clusters, production cluster
Hadoop in Action

This is an excerpt from Manning's book Hadoop in Action.

The output displays properties, such as permission, owner, group, file size, and last modification date, all of which are familiar Unix concepts. The column stating “1” reports the replication factor of the file. It should always be 1 for the pseudo-distributed configuration. For production clusters, the replication factor is typically 3 but can be any positive integer. Replication factor is not applicable to directories, so they will only show a dash (-) for that column.

Local mode has none of the distributed characteristics of a production Hadoop cluster. We may not see many bugs when running in local mode. Hadoop provides a pseudo-distributed mode that has all the functionality and “nodes” of a production cluster—Name Node, SecondaryNameNode, DataNode, JobTracker, and TaskTracker, each running on a separate JVM. All the software components are distributed, and pseudo-distributed mode differs from a production cluster at only the system-and-hardware level. It uses only one physical machine—your own local computer. We should make sure our jobs can run in pseudo-distributed mode before deploying them to a full production cluster.

Chapter 2 describes the configuration and commands to start pseudo-distributed mode. You’ll start all the daemons on your computer to make it function like a cluster. You interface with it as if it is a distinct Hadoop cluster. You put data into its own HDFS filesystem. You submit your jobs to it for running rather than run them in the same user space. Most importantly, you now monitor it “remotely” through log files and the web interface. You’ll use the same tools later to monitor a production cluster.

6.2. Monitoring and debugging on a production cluster

After successfully running your job in a pseudo-distributed cluster, you’re ready to run it on a production cluster using real data. We can apply all the techniques we’ve used for development and debugging on the production cluster, although the exact usage may be slightly different. Your cluster should still have a JobTracker Web UI, but the domain is no longer localhost. It’s now the address of the cluster’s JobTracker. The port number will still be 50030 unless it’s been configured differently.