Rails下cloud datastore的使用

 

 Rails下cloud datastore的使用

背景

部门有一个项目要用Ruby做 WebAPI，DB使用关系型数据库Cloud Sql和非关系型数据库Cloud Datastore 。

还不了解Ruby On Rails和CloudDatastore的请参考下面的链接。

http://guides.ruby-china.org/

https://thinkit.co.jp/story/2015/02/05/5594

 

1、 Windows下开发环境构建

1.1、构建Ruby On Rails 开发环境

① Ruby安装

http://www.runoob.com/ruby/ruby-installation-windows.html

② Rail安装

从CMD提示窗口输入指令：gem install rails 开始安装rails。

注意：在中国直接安装会提示错误，因为gem默认的安装源https://rubygems.org在国内不能访问，还好先辈们早已为我们新搭建了一个gem安装源http://gems.ruby-china.org。下面是切换安装源的具体步骤，gem安装源切换之后，Rails安装就可以顺利进行了。

第一步：删除默认安装源

CMD提示窗中输入”gem sources -remove https://rubygems.org/”，回车换行。

第二步：添加新的安装源

CMD提示窗输入“gem sources -add  http://gems.ruby-china.org/”，回车换行。

③ ruby的IDE Ruby Mine安装

http://qiita.com/HALU5071/items/6d6a39e44865d8d04de8

1.2、搭建开发用Cloud Datastore 数据库

① 安装Google Cloud SDK

http://www.apps-gcp.com/google-cloud-sdk-install

② 安装cloud-datastore-emulator

CMD提示窗中输入“gcloud components install cloud-datastore-emulator”，回车换行。

③ 创建一个开发用的DB:dev_datastore

CMD提示窗中输入“cloud_datastore_emulator create db/dev_datastore”,回车换行。

前提条件：CMD当前运行目录下要先创建一个db的文件夹。

创建好的DB文件结构图如下：

 

2、 WEB API工程创建

在CMD提示窗中输入“rails new my-first-api --api”，回车换行。

创建好的API工程结构见下图，rails执行结果见附录1。

フォルダ構造
		説明
Gemfile		gemの依存関係を指定できるファイル
README.rdoc		説明書
Rakefile	ターミナルから実行可能なタスク
config.ru		Rackの設定
app/		アプリケーションを格納するディレクトリ
		主要なプログラムはこの配下に格納
	controllers/	コントローラを格納するディレクトリ
	controllers/application_controller.rb	アプリケーションで共通のコントローラ
	models/	モデルを格納するディレクトリ
config/		プロジェクトの設定ファイルを格納するディレクトリ
config/environments/		環境単位の設定ファイルを格納するディレクトリ
config/initializers/		初期化ファイルを格納するディレクトリ
config/locales/		辞書ファイルを格納するディレクトリ
db/		データベースの設定ファイルを格納するディレクトリ
doc/		ドキュメントを格納するディレクトリ
lib/		複数のアプリケーション間で共有するライブラリを格納するディレクトリ
	tasks/	自分で生成したRakefileを格納するディレクトリ
log/		ログファイルが格納されるディレクトリ。ログファイルはアプリケーションと環境ごとに作成される
public/		Web上に公開するファイルを格納するディレクトリ
tmp/		キャッシュなど、一時的なファイルを格納されるディレクトリ
test/		アプリケーションのテストに使うファイルを格納するディレクトリ
vendor/		ライブラリや他のアプリケーションで共有するような外部ライブラリを格納するディレクトリ

 

在附录1的最后我们看到，bundle install 并没有成功。原因是项目的gem安装源出了问题。打开Gemfile把第一行的source 'https://rubygems.org'替换成source 'http://gems.ruby-china.org'。然后在CMD里重新执行bundle install命令，这样项目需要的gem就能成功安装。

 

3、 新建cloud Datastore 的dataset对象

① Gem安装

在Gemfile 中追加

gem 'google-cloud-datastore'

cmd里重新执行bundle install命令。

② 配置文件

在config/initializers/目录下新建cloud_datastore.rb文件，文件内容如下图所示。

从图中我们可以看到cloud datastore 配置了三种环境下创建daset的参数，这些参数我们都放在了config/database.yml里。下面是database.yml里开发环境的配置信息，datastore 安装的主机和datastore的数据库名，这样我们就可以使用datastore了。

development:
        host: 'localhost:8180'
        project: ' dev_datastore '

 

 

cloud_datastore.rb

 

4、 非关系型数据库 中ActiveModel 的使用

如果我们想要cloud datastore和关系型数据库一样，可以方便快捷使用model（ActiveModel），怎么办？下面就是你想要的答案。

① 追加文件

在config/initializers/目录下新建active_model_cloud_datastore.rb文件。

文件内容参照附录2。

② model Class写法

require_relative "../../config/initializers/active_model_cloud_datastore" 

class Customer
include ActiveModelCloudDatastore
attr_accessor :customer_code, :customer_name

def attributes
%w(code customer_name)# 注意多个字段之间是半角空格来区分的
end

end

这样我们就可以使用activeModel很多功能，

#检索所有数据

@customers = Customer.all

#保存一条新的数据

@customer = Customer.new

@customer.save

这里就不做详细说明，具体参照附录2里的各种方法。

5、 spec测试

1. 1.   
2. 2.   
3. 3.   
4. 4.   
5. 5.   

① 测试环境安装

在Gemfile中追加下面的gem。

group :development, :test do 
# Test
gem 'rspec-rails', '~> 3.0'
gem 'rails-controller-testing'
end

然后在cmd工程目录my-first-api下执行bundle install，安装追加的gem。

② 测试环境初期化

在cmd工程目录my-first-api下执行

bundle exec rails generate rspec:install

会生成下面的文件

create  .rspec

create  spec

create  spec/spec_helper.rb

create  spec/rails_helper.rb

③ controller测试

参考下面链接

http://qiita.com/shizuma/items/84e07e558abd6593df15

http://blog.naichilab.com/entry/2016/01/19/011514

④ model测试

参考下面的测试思路。

http://qiita.com/shizuma/items/c7b8d7b91e8f325f8ad9

⑤ Mock使用

参考下面链接

http://qiita.com/jnchito/items/640f17e124ab263a54dd

6、 测试覆盖率报告

①  环境搭建

在Gemfile中追加下面的gem。

group :development, :test do
gem 'simplecov'
end

然后在cmd工程目录my-first-api下执行bundle install，安装追加的gem。

② spec_helper.rb修改

在spec_helper.rb头部插入下面的语句，覆盖率测试中要过滤掉spec下的测试代码。

require 'simplecov'
SimpleCov.start do
add_filter "/spec/ "
end

③ 查看报告

在cmd工程目录my-first-api下，再次执行rspec spec命令，覆盖率报告就会自动生成再demo/coverage下，用google chrome浏览器打开index.html，就可以看到详细的信息。下面是一个覆盖率报告的截图。

7、 结束语

上面我们讲述的是Ruby下怎么使用cloud datastore的开发和测试，在Google Cloud Platform上怎么部署产品还有待下一步探索。期间遇到的各种技术问题难题，为了解决这些问题，调查的网站以日语和英语为主，总结的时候也使用了很多日语网站，由于时间有限，没能一一翻译过来，给不懂日语的朋友带来不少困难表示歉意。

附录1

API工程创建文件list

create

create  README.md

create  Rakefile

create  config.ru

create  .gitignore

create  Gemfile

create  app

create  app/assets/config/manifest.js

create  app/assets/javascripts/application.js

create  app/assets/javascripts/cable.js

create  app/assets/stylesheets/application.css

create  app/channels/application_cable/channel.rb

create  app/channels/application_cable/connection.rb

create  app/controllers/application_controller.rb

create  app/helpers/application_helper.rb

create  app/jobs/application_job.rb

create  app/mailers/application_mailer.rb

create  app/models/application_record.rb

create  app/views/layouts/application.html.erb

create  app/views/layouts/mailer.html.erb

create  app/views/layouts/mailer.text.erb

create  app/assets/images/.keep

create  app/assets/javascripts/channels

create  app/assets/javascripts/channels/.keep

create  app/controllers/concerns/.keep

create  app/models/concerns/.keep

create  bin

create  bin/bundle

create  bin/rails

create  bin/rake

create  bin/setup

create  bin/update

create  config

create  config/routes.rb

create  config/application.rb

create  config/environment.rb

create  config/secrets.yml

create  config/cable.yml

create  config/puma.rb

create  config/environments

create  config/environments/development.rb

create  config/environments/production.rb

create  config/environments/test.rb

create  config/initializers

create  config/initializers/application_controller_renderer.rb

create  config/initializers/assets.rb

create  config/initializers/backtrace_silencers.rb

create  config/initializers/cookies_serializer.rb

create  config/initializers/cors.rb

create  config/initializers/filter_parameter_logging.rb

create  config/initializers/inflections.rb

create  config/initializers/mime_types.rb

create  config/initializers/new_framework_defaults.rb

create  config/initializers/session_store.rb

create  config/initializers/wrap_parameters.rb

create  config/locales

create  config/locales/en.yml

create  config/boot.rb

create  config/database.yml

create  db

create  db/seeds.rb

create  lib

create  lib/tasks

create  lib/tasks/.keep

create  lib/assets

create  lib/assets/.keep

create  log

create  log/.keep

create  public

create  public/404.html

create  public/422.html

create  public/500.html

create  public/apple-touch-icon-precomposed.png

create  public/apple-touch-icon.png

create  public/favicon.ico

create  public/robots.txt

create  test/fixtures

create  test/fixtures/.keep

create  test/fixtures/files

create  test/fixtures/files/.keep

create  test/controllers

create  test/controllers/.keep

create  test/mailers

create  test/mailers/.keep

create  test/models

create  test/models/.keep

create  test/helpers

create  test/helpers/.keep

create  test/integration

create  test/integration/.keep

create  test/test_helper.rb

create  tmp

create  tmp/.keep

create  tmp/cache

create  tmp/cache/assets

create  vendor/assets/stylesheets

create  vendor/assets/stylesheets/.keep

remove  app/assets

remove  lib/assets

remove  tmp/cache/assets

remove  vendor/assets

remove  app/helpers

remove  test/helpers

remove  app/views/layouts/application.html.erb

remove  public/404.html

remove  public/422.html

remove  public/500.html

remove  public/apple-touch-icon-precomposed.png

remove  public/apple-touch-icon.png

remove  public/favicon.ico

remove  app/assets/javascripts

remove  config/initializers/assets.rb

remove  config/initializers/session_store.rb

remove  config/initializers/cookies_serializer.rb

Fetching gem metadata from https://rubygems.org/..........

Fetching version metadata from https://rubygems.org/..

Fetching dependency metadata from https://rubygems.org/.

Resolving dependencies...

Using rake 12.0.0

Using i18n 0.7.0

Using minitest 5.10.1

Using thread_safe 0.3.5

Using builder 3.2.2

Using erubis 2.7.0

Using mini_portile2 2.1.0

Using rack 2.0.1

Using nio4r 1.2.1

Using websocket-extensions 0.1.2

Using mime-types-data 3.2016.0521

Using arel 7.1.4

Using bundler 1.13.6

Using method_source 0.8.2

Using puma 3.6.2

Using thor 0.19.4

Using sqlite3 1.3.12

Gem::RemoteFetcher::FetchError: SSL_connect returned=1 errno=0 state=SSLv3 read

server certificate B: certificate verify failed

(https://rubygems.org/gems/concurrent-ruby-1.0.4.gem)

An error occurred while installing concurrent-ruby (1.0.4), and Bundler cannot

continue.

Make sure that `gem install concurrent-ruby -v '1.0.4'` succeeds before

bundling.

附录2

active_model_cloud_datastore.rb文件内容：

# frozen_string_literal: true

require_relative 'cloud_datastore'
require 'active_model'
require 'active_support'


# Integrates ActiveModel with the Google::Cloud::Datastore
module ActiveModelCloudDatastore
extend ActiveSupport::Concern
include ActiveModel::Model
include ActiveModel::Dirty
include ActiveModel::Validations
include ActiveModel::Validations::Callbacks

included do
private_class_method :query_options, :query_sort, :query_property_filter
define_model_callbacks :save, :update, :destroy
attr_accessor :id
end

def attributes
[]
end

# Used by ActiveModel for determining polymorphic routing.
def persisted?
id.present?
end

# Resets the ActiveModel::Dirty tracked changes
def reload!
clear_changes_information
end

# Updates attribute values on the ActiveModel::Model object with the provided params.
# Example, such as submitted form params.
#
# @param [Hash] params
def update_model_attributes(params)
params.each do |name, value|
send "#{name}=", value if respond_to? "#{name}="
end
end

# Builds the Cloud Datastore entity with attributes from the Model object.
#
# @return [Entity] the updated Google::Cloud::Datastore::Entity
def build_entity(parent = nil)
entity = CloudDatastore.dataset.entity(self.class.name, id)
entity.key.parent = parent if parent
attributes.each do |attr|
entity[attr] = instance_variable_get("@#{attr}")
end
entity
end

def save(parent = nil)
run_callbacks :save do
if valid?
entity = build_entity(parent)
success = self.class.retry_on_exception? { CloudDatastore.dataset.save(entity) }
if success
self.id = entity.key.id
return true
end
end
false
end
end

def update(params)
run_callbacks :update do
update_model_attributes(params)
if valid?
entity = build_entity
self.class.retry_on_exception? { CloudDatastore.dataset.save(entity) }
else
false
end
end
end

def destroy
run_callbacks :destroy do
key = CloudDatastore.dataset.key(self.class.name, id)
self.class.retry_on_exception? { CloudDatastore.dataset.delete(key) }
end
end

# Methods defined here will be class methods whenever we 'include DatastoreUtils'.
module ClassMethods
# Queries all objects from Cloud Datastore by named kind and using the provided options.
#
# @param [Hash] options the options to construct the query with.
#
# @option options [Google::Cloud::Datastore::Key] :ancestor filter for inherited results
# @option options [Hash] :where filter, Array in the format [name, operator, value]
#
# @return [Array<Model>] an array of ActiveModel results.
def all(options = {})
query = CloudDatastore.dataset.query(name)
query.ancestor(options[:ancestor]) if options[:ancestor]
query_property_filter(query, options)
entities = retry_on_exception { CloudDatastore.dataset.run(query) }
from_entities(entities.flatten)
end

# Queries objects from Cloud Datastore in batches by named kind and using the provided options.
# When a limit option is provided queries up to the limit and returns results with a cursor.
#
# @param [Hash] options the options to construct the query with. See build_query for options.
#
# @return [Array<Model>, String] an array of ActiveModel results and a cursor that can be used
# to query for additional results.
def find_in_batches(options = {})
next_cursor = nil
query = build_query(options)
if options[:limit]
entities = retry_on_exception { CloudDatastore.dataset.run(query) }
next_cursor = entities.cursor if entities.size == options[:limit]
else
entities = retry_on_exception { CloudDatastore.dataset.run(query) }
end
model_entities = from_entities(entities.flatten)
return model_entities, next_cursor
end

# Retrieves an entity by key and by an optional parent.
#
# @param [Integer or String] id_or_name id or name value of the entity Key.
# @param [Google::Cloud::Datastore::Key] parent the parent Key of the entity.
#
# @return [Entity, nil] a Google::Cloud::Datastore::Entity object or nil.
def find_entity(id_or_name, parent = nil)
key = CloudDatastore.dataset.key(name, id_or_name)
key.parent = parent if parent
retry_on_exception { CloudDatastore.dataset.find(key) }
end

# Find object by ID.
#
# @return [Model, nil] an ActiveModel object or nil.
def find(id)
entity = find_entity(id.to_i)
from_entity(entity)
end

# Find object by parent and ID.
#
# @return [Model, nil] an ActiveModel object or nil.
def find_by_parent(id, parent)
entity = find_entity(id.to_i, parent)
from_entity(entity)
end

def from_entities(entities)
entities.map { |entity| from_entity(entity) }
end

# Translates between Google::Cloud::Datastore::Entity objects and ActiveModel::Model objects.
#
# @param [Entity] entity from Cloud Datastore
# @return [Model] the translated ActiveModel object.
def from_entity(entity)
return if entity.nil?
model_entity = new
model_entity.id = entity.key.id unless entity.key.id.nil?
model_entity.id = entity.key.name unless entity.key.name.nil?
entity.properties.to_hash.each do |name, value|
model_entity.send "#{name}=", value
end
model_entity.reload!
model_entity
end

def exclude_from_index(entity, boolean)
entity.properties.to_h.keys.each do |value|
entity.exclude_from_indexes! value, boolean
end
end

# Constructs a Google::Cloud::Datastore::Query.
#
# @param [Hash] options the options to construct the query with.
#
# @option options [Google::Cloud::Datastore::Key] :ancestor filter for inherited results
# @option options [String] :cursor sets the cursor to start the results at
# @option options [Integer] :limit sets a limit to the number of results to be returned
# @option options [String] :order sort the results by property name
# @option options [String] :desc_order sort the results by descending property name
# @option options [Array] :select retrieve only select properties from the matched entities
# @option options [Hash] :where filter, Array in the format [name, operator, value]
#
# @return [Query] a datastore query.
def build_query(options = {})
query = CloudDatastore.dataset.query(name)
query_options(query, options)
end

def retry_on_exception?
retry_count = 0
sleep_time = 0.5 # 0.5, 1, 2, 4 second between retries
begin
yield
rescue => e
puts "\e[33m[#{e.message.inspect}]\e[0m"
puts 'Rescued exception, retrying...'
sleep sleep_time
sleep_time *= 2
retry_count += 1
return false if retry_count > 3
retry
end
true
end

def retry_on_exception
retry_count = 0
sleep_time = 0.5 # 0.5, 1, 2, 4 second between retries
begin
yield
rescue => e
puts "\e[33m[#{e.message.inspect}]\e[0m"
puts 'Rescued exception, retrying...'
sleep sleep_time
sleep_time *= 2
retry_count += 1
raise e if retry_count > 3
retry
end
end

def log_google_cloud_error
yield
rescue Google::Cloud::Error => e
puts "\e[33m[#{e.message.inspect}]\e[0m"
raise e
end

# private

def query_options(query, options)
query.ancestor(options[:ancestor]) if options[:ancestor]
query.cursor(options[:cursor]) if options[:cursor]
query.limit(options[:limit]) if options[:limit]
query_sort(query, options)
query.select(options[:select]) if options[:select]
query_property_filter(query, options)
end

# Adds sorting to the results by a property name if included in the options.
def query_sort(query, options)
query.order(options[:order]) if options[:order]
query.order(options[:desc_order], :desc) if options[:desc_order]
query
end

# Adds property filters to the query if included in the options.
# Accepts individual or nested Arrays:
# [['superseded', '=', false], ['email', '=', 'something']]
def query_property_filter(query, options)
if options[:where]
opts = options[:where]
if opts[0].is_a?(Array)
opts.each do |opt|
query.where(opt[0], opt[1], opt[2]) unless opt.nil?
end
else
query.where(opts[0], opts[1], opts[2])
end
end
query
end
end

end