apache mesos

by Marco Massenzio

由Marco Massenzio

试用Apache Mesos HTTP API获得乐趣和收益 (Experimenting with the Apache Mesos HTTP API for Fun and Profit)

Apache Mesos is a tool used in production at large-scale services like Twitter and Airbnb. Here’s its textbook description:

Apache Mesos是Twitter和Airbnb等大规模服务中用于生产的工具。这是它的教科书描述：

The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elasticsearch) with API’s for resource management and scheduling across entire datacenter and cloud environments. — from the Apache Mesos project site.
Mesos内核在每台机器上运行，并为应用程序(例如Hadoop，Spark，Kafka，Elasticsearch)提供API，用于在整个数据中心和云环境中进行资源管理和调度。 —从Apache Mesos项目站点。

This is the first of a series of three articles that show how to setup a Vagrant-based Apache Mesos test/development environment on your laptop, how to run a Python notebook against the HTTP API, and how to launch Docker containers on the running Agent VM.

这是三篇系列文章中的第一篇，该系列文章介绍如何在笔记本电脑上设置基于Vagrant的Apache Mesos测试/开发环境，如何针对HTTP API运行Python笔记本以及如何在正在运行的Agent上启动Docker容器虚拟机。

This series is an extended (and updated) version of the talk I gave at MesosCon Europe 2015 updated for Apache Mesos 1.0.0, which has just been released (August 2016) — you can also find the slides there.

本系列是我在MesosCon Europe 2015上 演讲的扩展(和更新)版本，该演讲针对Apache Mesos 1.0.0(已于2016年8月发布)进行了更新-您也可以在此处找到幻灯片。

This post is pretty jam-packed, and will require you to be familiarity with some concepts around containers, VMs, and Mesos. But I’ll take the time to show all the intermediate steps (hence, the 3-parts). It should be easy to follow, even if you’ve never used Vagrant, Mesos, or even Jupyter notebooks before.

这篇文章挤满了人，需要您熟悉有关容器，VM和Mesos的一些概念。但是，我将花一些时间介绍所有中间步骤(因此分为3部分)。即使您以前从未使用过Vagrant，Mesos甚至Jupyter笔记本电脑，也应该易于遵循。

I recommend you first have a basic familiarity with Python and handling HTTP requests and responses, as we will not be going those details there.

我建议您首先对Python和处理HTTP请求和响应有一个基本的了解，因为我们在这里不再赘述。

All the code is available on the zk-mesos git repository:

所有代码都可以在zk-mesos git存储库中找到：

git clone git@github.com:massenz/zk-mesos.git

And you can also view the README.

您还可以查看README 。

入门 (Getting Started)

In order to follow along, you will need to clone the repository (as shown above) and install Virtualbox and Vagrant. Follow the instructions on their respective sites and you’ll be up and running in no time.

为了进行后续操作，您将需要克隆存储库(如上所示)并安装Virtualbox和Vagrant 。按照其各自站点上的说明进行操作，您将立即启动并运行。

I also recommend quickly scanning the Vagrant documentation. A knowledge of Vagrant beyond `vagrant up` is not really required to get the most out of this series, but it may help if you get stuck (or would like to experiment and improve on our Vagrantfile).

我还建议快速扫描Vagrant文档。要充分利用本系列的知识，并不是真正需要“ vagrant up ”以外的Vagrant知识，但是如果您陷入困境(或者想对我们的Vagrantfile进行试验和改进)，则可能会有所帮助。

If you’re not familiar with Apache Mesos I recommend taking a look at the project’s site. I recommend reading Mesos in Action (note that I was one of the manuscript’s reviewers).

如果您不熟悉Apache Mesos ，建议您查看该项目的站点。我建议阅读《 Mesos in Action》 (请注意，我是手稿的审阅者之一)。

We will not be building Mesos from source here, but will instead use Mesosphere packages. You don’t need to download them. The Vagrantfile will automatically download and install on the VMs.

我们不会在这里从源代码构建Mesos，而是使用Mesosphere软件包。您不需要下载它们。 Vagrantfile将自动下载并安装在VM上。

To run the Python notebook, we’ll take advantage of the Jupyter packages, and use a virtualenv to run all of our code. Virtualenv isn’t strictly necessary, but will prevent you messing up your system Python.

要运行Python笔记本，我们将利用Jupyter软件包，并使用virtualenv运行所有代码。 Virtualenv不是严格必需的，但是可以防止您弄乱系统Python。

If you‘ve never used virtualenv before:

如果您以前从未使用过virtualenv ：

$ sudo pip install virtualenv

And then create and run a virtualenv:

然后创建并运行一个virtualenv ：

$ cd zk-mesos $ virtualenv mesos-demo$ source mesos-demo/bin/activate $ pip install -r requirements.txt

Finally, verify that you can run and load the Jupyter notebook:

最后，确认您可以运行并加载Jupyter笔记本：

$ jupyter notebook

This should automatically open your browser and point it to http://localhost:8888. From here you can select the notebooks/Demo-API.ipynb file. Don’t run it just yet, but if it shows up, it will confirm that your Python setup is just fine.

这应该会自动打开浏览器，并将其指向http：// localhost：8888 。在这里，您可以选择notebooks / Demo-API.ipynb文件。暂时不要运行它，但是如果显示出来，它将确认您的Python设置很好。

构建和安装Apache Mesos (Building and installing Apache Mesos)

This is where the beauty of Vagrant shines in all its glory. Installing Apache Mesos Master and Agent are not trivial tasks, but in our case, it’s just a matter of:

这就是流浪者的美丽在所有荣耀中闪耀的地方。安装Apache Mesos Master和Agent并不是一件容易的事，但就我们而言，这只是一个问题：

$ cd vagrant $ vagrant up

Make sure that you’re in the same directory as the Vagrantfile when issuing any of the Vagrant commands, or it will complain about it.

发出任何Vagrant命令时，请确保与Vagrantfile位于同一目录中，否则它将抱怨。

It is worth noting that we are building two Vagrant boxes, so any command will operate on both unless specified. To avoid this, you can specify the name of the VM after the command. For example, to SSH onto the Agent:

值得注意的是，我们正在构建两个 Vagrant框，因此，除非指定，否则任何命令都将对这两个框进行操作。为避免这种情况，您可以在命令后指定VM的名称。例如，要SSH到代理上：

$ vagrant ssh agent

This should log you in on that box. From there, you can explore, experiment, and diagnose any issues.

这将使您登录该框。从那里，您可以探索，试验和诊断任何问题。

The vagrant up command will take some time to execute, but it should eventually lead your Virtualbox to have two VMs, named respectively mesos-master and mesos-agent. Incidentally, you should never need to use VirtualBox to manage them. All the tasks can be undertaken via Vagrant commands. But you can manage them manually if necessary or desired.

vagrant up命令将花费一些时间来执行，但最终将导致您的Virtualbox具有两个VM，分别命名为mesos-master和mesos-agent 。顺便说一句，您永远不需要使用VirtualBox来管理它们。所有任务都可以通过Vagrant命令执行。但是，如有必要或期望，您可以手动管理它们。

Once your VMs are built, ensure you can access Mesos HTTP UI at:

构建完虚拟机后，请确保可以通过以下方式访问Mesos HTTP UI：

http://192.168.33.10:5050

You should also see one agent running, accessible either via the Master UI.

您还应该看到一个正在运行的代理，可以通过主UI进行访问。

Or directly at:

或直接在：

http://192.168.33.11:5051/state

Note that the Agent runs not only on a different IP address than the Master, but also on a different port (5051 instead of 5050).

请注意，代理不仅在与主服务器不同的IP地址上运行，而且在不同的端口(5051而不是5050)上运行。

Look into vagrant/run-agent.sh to see a few of the command line flags that we use to run the Agent (and in run-master.sh for the Master).

查看vagrant / run-agent.sh，以查看我们用于运行代理的一些命令行标志(对于Master，在run-master.sh中)。

动物园管理员 (Zookeeper)

It’s worth noting that we are also running an instance of Zookeeper (for Leader election and Master/Agent coordination) on the mesos-master VM, inside a Docker container: partly because we can, but also to show how easy it is to do so using containers.

值得注意的是，我们还在Docker容器内的mesos-master VM上运行了Zookeeper实例(用于Leader选举和Master / Agent协调)：部分原因是我们可以这样做，但同时也表明这样做很容易使用容器。

This one line (in run-master.sh), will give you a perfectly good ZK instance (albeit, a catastrophically unreliable one in a production environment, where you’d want to run at least 3–5 nodes, at least, on physically separate machines/racks):

这一行(在run-master.sh中 )将为您提供一个非常好的ZK实例(尽管在生产环境中，这是一个灾难性的不可靠实例)，在该生产环境中，您至少要在3-5个节点上运行物理上分开的机器/机架)：

docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 \    -p 3888:3888 jplock/zookeeper:3.4.8

And because we expose the ports (in particular, 2181) to the host VM, we can connect to it via the Zookeeper CLI utility (zkCli.sh) and explore it. From your development machine (you will need to first download Zookeeper) you can use:

并且由于我们将端口(特别是2181)公开给主机VM，因此我们可以通过Zookeeper CLI实用程序( zkCli.sh )连接到它并进行浏览。在开发机器(您需要首先下载Zookeeper)中，可以使用：

$ zkCli.sh -server 192.168.33.10:2181...[zk: 192.168.33.10:2181(CONNECTED) 4] get /mesos/vagrant/json.info_0000000000

# Formatted for better readability:{"address":  { "hostname": "mesos-master",    "ip":"192.168.33.10",    "port":5050  },  "hostname":"mesos-master",  "id":"7eb34f10-b07c-4921-aece-bbaece09dfd1",  "ip":169978048,  "pid":"master@192.168.33.10:5050",  "port":5050,  "version":"1.0.0"}

cZxid = 0xbctime = Sat Aug 27 14:00:44 PDT 2016...

This is how Agents get information about how to connect to the Master node.

这是代理如何获取有关如何连接到主节点的信息。

The _000000 suffix gets incremented every time a new Leader gets elected, so depending on how long the ZK instance has been running and whether the Master was restarted, it may become something like _0000005. That is an “ephemeral node” in Zookeeper’s parlance.

每次选举新的Leader时，后缀_000000都会增加，因此取决于ZK实例运行了多长时间以及是否重新启动了主服务器，它的后缀可能类似于_0000005。 在Zookeeper的说法中，这是一个“ 短暂节点” 。

In that record above, it is worth noting that “pid” is libprocess unique identifier and “ip” as a compressed 4-byte representation of an IPv4 octect quadruple. These legacy fields and may eventually be removed.

在上面的记录中，值得注意的是，“ pid”是libprocess唯一标识符，“ ip”是IPv4 octect四元组的压缩4字节表示形式。这些旧字段可能最终会被删除。