原文: medium.com/@amsokol.co…
关于如何使用一些优秀的框架或者路由来编写Go REST微服务了已经有很多文章了,当我为我司寻找合适的服务构建方法时,我大量地阅读了它们。突然间我发现一个非常有趣的方法去构建HTTP/REST微服务,就是用Google开源的protobuf/gRPC框架。我确信已经有很多人听说过它,甚至有一部分人已经正在使用。但是我相信实际很少人有使用protobuf/gRPC框架来开发HTTP/REST微服务的经验。我只找到一篇 文章(需要翻)在Medium上。
在此我并不打算重复它的内容。我希望通过一步步的引导让大家学会如何去开发一个简单的CRUD “To Do List”的微服务通过使用gPRC以及HTTP/REST的后端接口。我会演示如何编写测试用例以及加入中间件(请求ID以及日志记录与追踪)在服务当中。最后甚至还会提供一些例子讲述如何构建以及发布我们的微服务到Kubernetes上。
整个教程会分为4个部分:
这句话有什么意义?
“To Do List”微服务允许去管理“To Do”列表,ToDo项包括以下字段/属性:
ToDo服务包含典型的增删改查以及获取全部项方法。
Part1完整代码 戳这里
开始之前我们先构建一个项目的骨架, 这里 又一个非常优秀的Go项目的脚手架模板
我是用的是Windows 10 x64运行环境,但是我想你将我接下来的cmd指令转换成MacOs/Linux BASH应该不是什么大问题
首先创建文件夹并初始化项目
mkdir go-grpc-http-rest-microservice-tutorial cd go-grpc-http-rest-microservice-tutorial go mod init github.com/<you>/go-grpc-http-rest-microservice-tutorial 复制代码
在你的项目里面创建文件目录如下
mkdir -p api/proto/v1 复制代码
这里的v1就是我们API的版本号
API版本化:通过将不同版本的API代码放到不同文件夹里面并以此命名是我的最佳实践
下一步就是在刚创建的 v1 文件夹里面创建 todo-service.proto 文件并加入ToDo服务的定义,我们先从Create方法写起:
syntax = "proto3"; package v1; import "google/protobuf/timestamp.proto"; // Taks we have to do message ToDo { // Unique integer identifier of the todo task int64 id = 1; // Title of the task string title = 2; // Detail description of the todo task string description = 3; // Date and time to remind the todo task google.protobuf.Timestamp reminder = 4; } // Request data to create new todo task message CreateRequest{ // API versioning: it is my best practice to specify version explicitly string api = 1; // Task entity to add ToDo toDo = 2; } // Response that contains data for created todo task message CreateResponse{ // API versioning: it is my best practice to specify version explicitly string api = 1; // ID of created task int64 id = 2; } // Service to manage list of todo tasks service ToDoService { // Create new todo task rpc Create(CreateRequest) returns (CreateResponse); } 复制代码
戳我 查看Proto的编写语法
如你所见,API定义是跟编程语言,通讯协议以及网络传输无关的,这也是protobuf的一个重要标志
为了能编译proto文件我们需要安装一些工具和依赖
go get -u github.com/golang/protobuf/protoc-gen-go 复制代码
# Windows: ./third_party/protoc-gen.cmd # MasOS/Linux: ./third_party/protoc-gen.sh 复制代码
运行以后会创建一个名为 todo-service.pb.go 的文件在 pkg/model/v1 下
接下来让我们将剩下的方法驾到ToDo服务当中并编译
syntax = "proto3"; package v1; import "google/protobuf/timestamp.proto"; // Taks we have to do message ToDo { // Unique integer identifier of the todo task int64 id = 1; // Title of the task string title = 2; // Detail description of the todo task string description = 3; // Date and time to remind the todo task google.protobuf.Timestamp reminder = 4; } // Request data to create new todo task message CreateRequest{ // API versioning: it is my best practice to specify version explicitly string api = 1; // Task entity to add ToDo toDo = 2; } // Contains data of created todo task message CreateResponse{ // API versioning: it is my best practice to specify version explicitly string api = 1; // ID of created task int64 id = 2; } // Request data to read todo task message ReadRequest{ // API versioning: it is my best practice to specify version explicitly string api = 1; // Unique integer identifier of the todo task int64 id = 2; } // Contains todo task data specified in by ID request message ReadResponse{ // API versioning: it is my best practice to specify version explicitly string api = 1; // Task entity read by ID ToDo toDo = 2; } // Request data to update todo task message UpdateRequest{ // API versioning: it is my best practice to specify version explicitly string api = 1; // Task entity to update ToDo toDo = 2; } // Contains status of update operation message UpdateResponse{ // API versioning: it is my best practice to specify version explicitly string api = 1; // Contains number of entities have beed updated // Equals 1 in case of succesfull update int64 updated = 2; } // Request data to delete todo task message DeleteRequest{ // API versioning: it is my best practice to specify version explicitly string api = 1; // Unique integer identifier of the todo task to delete int64 id = 2; } // Contains status of delete operation message DeleteResponse{ // API versioning: it is my best practice to specify version explicitly string api = 1; // Contains number of entities have beed deleted // Equals 1 in case of succesfull delete int64 deleted = 2; } // Request data to read all todo task message ReadAllRequest{ // API versioning: it is my best practice to specify version explicitly string api = 1; } // Contains list of all todo tasks message ReadAllResponse{ // API versioning: it is my best practice to specify version explicitly string api = 1; // List of all todo tasks repeated ToDo toDos = 2; } // Service to manage list of todo tasks service ToDoService { // Create new todo task rpc Create(CreateRequest) returns (CreateResponse); // Read todo task rpc Read(ReadRequest) returns (ReadResponse); // Update todo task rpc Update(UpdateRequest) returns (UpdateResponse); // Delete todo task rpc Delete(DeleteRequest) returns (DeleteResponse); // Read all todo tasks rpc ReadAll(ReadAllRequest) returns (ReadAllResponse); } 复制代码
再一次运行以下命令来更新Go的代码
# Windows: ./third_party/protoc-gen.cmd # MasOS/Linux: ./third_party/protoc-gen.sh 复制代码
到此为止,API的定义就完成了
我是使用Google Cloud上的MySQL作为数据库来持久化做存储的。你可以使用其他你喜欢的SQL数据库。
创建ToDo table的MySQL脚本
CREATE TABLE `ToDo` ( `ID` bigint(20) NOT NULL AUTO_INCREMENT, `Title` varchar(200) DEFAULT NULL, `Description` varchar(1024) DEFAULT NULL, `Reminder` timestamp NULL DEFAULT NULL, PRIMARY KEY (`ID`), UNIQUE KEY `ID_UNIQUE` (`ID`) ); 复制代码
我会跳过如何安装配置SQL数据库以及创建数据表的步骤
创建文件 pkg/service/v1/todo-service.go 以及下面的内容
package v1 import ( "context" "database/sql" "fmt" "time" "github.com/golang/protobuf/ptypes" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" "github.com/amsokol/go-grpc-http-rest-microservice-tutorial/pkg/api/v1" ) const ( // apiVersion is version of API is provided by server apiVersion = "v1" ) // toDoServiceServer is implementation of v1.ToDoServiceServer proto interface type toDoServiceServer struct { db *sql.DB } // NewToDoServiceServer creates ToDo service func NewToDoServiceServer(db *sql.DB) v1.ToDoServiceServer { return &toDoServiceServer{db: db} } // checkAPI checks if the API version requested by client is supported by server func (s *toDoServiceServer) checkAPI(api string) error { // API version is "" means use current version of the service if len(api) > 0 { if apiVersion != api { return status.Errorf(codes.Unimplemented, "unsupported API version: service implements API version '%s', but asked for '%s'", apiVersion, api) } } return nil } // connect returns SQL database connection from the pool func (s *toDoServiceServer) connect(ctx context.Context) (*sql.Conn, error) { c, err := s.db.Conn(ctx) if err != nil { return nil, status.Error(codes.Unknown, "failed to connect to database-> "+err.Error()) } return c, nil } // Create new todo task func (s *toDoServiceServer) Create(ctx context.Context, req *v1.CreateRequest) (*v1.CreateResponse, error) { // check if the API version requested by client is supported by server if err := s.checkAPI(req.Api); err != nil { return nil, err } // get SQL connection from pool c, err := s.connect(ctx) if err != nil { return nil, err } defer c.Close() reminder, err := ptypes.Timestamp(req.ToDo.Reminder) if err != nil { return nil, status.Error(codes.InvalidArgument, "reminder field has invalid format-> "+err.Error()) } // insert ToDo entity data res, err := c.ExecContext(ctx, "INSERT INTO ToDo(`Title`, `Description`, `Reminder`) VALUES(?, ?, ?)", req.ToDo.Title, req.ToDo.Description, reminder) if err != nil { return nil, status.Error(codes.Unknown, "failed to insert into ToDo-> "+err.Error()) } // get ID of creates ToDo id, err := res.LastInsertId() if err != nil { return nil, status.Error(codes.Unknown, "failed to retrieve id for created ToDo-> "+err.Error()) } return &v1.CreateResponse{ Api: apiVersion, Id: id, }, nil } // Read todo task func (s *toDoServiceServer) Read(ctx context.Context, req *v1.ReadRequest) (*v1.ReadResponse, error) { // check if the API version requested by client is supported by server if err := s.checkAPI(req.Api); err != nil { return nil, err } // get SQL connection from pool c, err := s.connect(ctx) if err != nil { return nil, err } defer c.Close() // query ToDo by ID rows, err := c.QueryContext(ctx, "SELECT `ID`, `Title`, `Description`, `Reminder` FROM ToDo WHERE `ID`=?", req.Id) if err != nil { return nil, status.Error(codes.Unknown, "failed to select from ToDo-> "+err.Error()) } defer rows.Close() if !rows.Next() { if err := rows.Err(); err != nil { return nil, status.Error(codes.Unknown, "failed to retrieve data from ToDo-> "+err.Error()) } return nil, status.Error(codes.NotFound, fmt.Sprintf("ToDo with ID='%d' is not found", req.Id)) } // get ToDo data var td v1.ToDo var reminder time.Time if err := rows.Scan(&td.Id, &td.Title, &td.Description, &reminder); err != nil { return nil, status.Error(codes.Unknown, "failed to retrieve field values from ToDo row-> "+err.Error()) } td.Reminder, err = ptypes.TimestampProto(reminder) if err != nil { return nil, status.Error(codes.Unknown, "reminder field has invalid format-> "+err.Error()) } if rows.Next() { return nil, status.Error(codes.Unknown, fmt.Sprintf("found multiple ToDo rows with ID='%d'", req.Id)) } return &v1.ReadResponse{ Api: apiVersion, ToDo: &td, }, nil } // Update todo task func (s *toDoServiceServer) Update(ctx context.Context, req *v1.UpdateRequest) (*v1.UpdateResponse, error) { // check if the API version requested by client is supported by server if err := s.checkAPI(req.Api); err != nil { return nil, err } // get SQL connection from pool c, err := s.connect(ctx) if err != nil { return nil, err } defer c.Close() reminder, err := ptypes.Timestamp(req.ToDo.Reminder) if err != nil { return nil, status.Error(codes.InvalidArgument, "reminder field has invalid format-> "+err.Error()) } // update ToDo res, err := c.ExecContext(ctx, "UPDATE ToDo SET `Title`=?, `Description`=?, `Reminder`=? WHERE `ID`=?", req.ToDo.Title, req.ToDo.Description, reminder, req.ToDo.Id) if err != nil { return nil, status.Error(codes.Unknown, "failed to update ToDo-> "+err.Error()) } rows, err := res.RowsAffected() if err != nil { return nil, status.Error(codes.Unknown, "failed to retrieve rows affected value-> "+err.Error()) } if rows == 0 { return nil, status.Error(codes.NotFound, fmt.Sprintf("ToDo with ID='%d' is not found", req.ToDo.Id)) } return &v1.UpdateResponse{ Api: apiVersion, Updated: rows, }, nil } // Delete todo task func (s *toDoServiceServer) Delete(ctx context.Context, req *v1.DeleteRequest) (*v1.DeleteResponse, error) { // check if the API version requested by client is supported by server if err := s.checkAPI(req.Api); err != nil { return nil, err } // get SQL connection from pool c, err := s.connect(ctx) if err != nil { return nil, err } defer c.Close() // delete ToDo res, err := c.ExecContext(ctx, "DELETE FROM ToDo WHERE `ID`=?", req.Id) if err != nil { return nil, status.Error(codes.Unknown, "failed to delete ToDo-> "+err.Error()) } rows, err := res.RowsAffected() if err != nil { return nil, status.Error(codes.Unknown, "failed to retrieve rows affected value-> "+err.Error()) } if rows == 0 { return nil, status.Error(codes.NotFound, fmt.Sprintf("ToDo with ID='%d' is not found", req.Id)) } return &v1.DeleteResponse{ Api: apiVersion, Deleted: rows, }, nil } // Read all todo tasks func (s *toDoServiceServer) ReadAll(ctx context.Context, req *v1.ReadAllRequest) (*v1.ReadAllResponse, error) { // check if the API version requested by client is supported by server if err := s.checkAPI(req.Api); err != nil { return nil, err } // get SQL connection from pool c, err := s.connect(ctx) if err != nil { return nil, err } defer c.Close() // get ToDo list rows, err := c.QueryContext(ctx, "SELECT `ID`, `Title`, `Description`, `Reminder` FROM ToDo") if err != nil { return nil, status.Error(codes.Unknown, "failed to select from ToDo-> "+err.Error()) } defer rows.Close() var reminder time.Time list := []*v1.ToDo{} for rows.Next() { td := new(v1.ToDo) if err := rows.Scan(&td.Id, &td.Title, &td.Description, &reminder); err != nil { return nil, status.Error(codes.Unknown, "failed to retrieve field values from ToDo row-> "+err.Error()) } td.Reminder, err = ptypes.TimestampProto(reminder) if err != nil { return nil, status.Error(codes.Unknown, "reminder field has invalid format-> "+err.Error()) } list = append(list, td) } if err := rows.Err(); err != nil { return nil, status.Error(codes.Unknown, "failed to retrieve data from ToDo-> "+err.Error()) } return &v1.ReadAllResponse{ Api: apiVersion, ToDos: list, }, nil } 复制代码
为API逻辑实现创建文件 pkg/service/v1/todo-service.go 最后文件目录如下
不管我们开发什么都应该编写测试用例。这是 强制遵守 的规定。
这里有一个很棒的模拟库,用于测试SQL数据库的交互 go-sqlmock .我将会使用它为我们的ToDo服务编写测试用例。
将 这个文件 放到 pkg/service/v1 目录下,当前项目文件结构如下
创建文件 pkg/protocol/grpc/server.go 并写入
package grpc import ( "context" "log" "net" "os" "os/signal" "google.golang.org/grpc" "github.com/amsokol/go-grpc-http-rest-microservice-tutorial/pkg/api/v1" ) // RunServer runs gRPC service to publish ToDo service func RunServer(ctx context.Context, v1API v1.ToDoServiceServer, port string) error { listen, err := net.Listen("tcp", ":"+port) if err != nil { return err } // register service server := grpc.NewServer() v1.RegisterToDoServiceServer(server, v1API) // graceful shutdown c := make(chan os.Signal, 1) signal.Notify(c, os.Interrupt) go func() { for range c { // sig is a ^C, handle it log.Println("shutting down gRPC server...") server.GracefulStop() <-ctx.Done() } }() // start gRPC server log.Println("starting gRPC server...") return server.Serve(listen) } 复制代码
RunServer函数负责注册ToDo服务以及启动gRPC服务
你需要给gPRC服务配置TLS,查看 示例 学习如何配置
接下来创建 pkg/cmd/server/server.go 以及对应内容
package cmd import ( "context" "database/sql" "flag" "fmt" // mysql driver _ "github.com/go-sql-driver/mysql" "github.com/amsokol/go-grpc-http-rest-microservice-tutorial/pkg/protocol/grpc" "github.com/amsokol/go-grpc-http-rest-microservice-tutorial/pkg/service/v1" ) // Config is configuration for Server type Config struct { // gRPC server start parameters section // gRPC is TCP port to listen by gRPC server GRPCPort string // DB Datastore parameters section // DatastoreDBHost is host of database DatastoreDBHost string // DatastoreDBUser is username to connect to database DatastoreDBUser string // DatastoreDBPassword password to connect to database DatastoreDBPassword string // DatastoreDBSchema is schema of database DatastoreDBSchema string } // RunServer runs gRPC server and HTTP gateway func RunServer() error { ctx := context.Background() // get configuration var cfg Config flag.StringVar(&cfg.GRPCPort, "grpc-port", "", "gRPC port to bind") flag.StringVar(&cfg.DatastoreDBHost, "db-host", "", "Database host") flag.StringVar(&cfg.DatastoreDBUser, "db-user", "", "Database user") flag.StringVar(&cfg.DatastoreDBPassword, "db-password", "", "Database password") flag.StringVar(&cfg.DatastoreDBSchema, "db-schema", "", "Database schema") flag.Parse() if len(cfg.GRPCPort) == 0 { return fmt.Errorf("invalid TCP port for gRPC server: '%s'", cfg.GRPCPort) } // add MySQL driver specific parameter to parse date/time // Drop it for another database param := "parseTime=true" dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s?%s", cfg.DatastoreDBUser, cfg.DatastoreDBPassword, cfg.DatastoreDBHost, cfg.DatastoreDBSchema, param) db, err := sql.Open("mysql", dsn) if err != nil { return fmt.Errorf("failed to open database: %v", err) } defer db.Close() v1API := v1.NewToDoServiceServer(db) return grpc.RunServer(ctx, v1API, cfg.GRPCPort) } 复制代码
RunServer函数负责读取命令行输入的参数,创建数据库连接,创建ToDo服务实例以及调用之前gPRC服务中的 RunServer 函数
最后创建以下文件 cmd/server/main.go 以及内容
package main import ( "fmt" "os" "github.com/amsokol/go-grpc-http-rest-microservice-tutorial/pkg/cmd" ) func main() { if err := cmd.RunServer(); err != nil { fmt.Fprintf(os.Stderr, "%v/n", err) os.Exit(1) } } 复制代码
以上就是服务端所有的代码了,当前项目目录如下
创建文件 cmd/client-grpc/main.go 以及以下内容
package main import ( "context" "flag" "log" "time" "github.com/golang/protobuf/ptypes" "google.golang.org/grpc" "github.com/amsokol/go-grpc-http-rest-microservice-tutorial/pkg/api/v1" ) const ( // apiVersion is version of API is provided by server apiVersion = "v1" ) func main() { // get configuration address := flag.String("server", "", "gRPC server in format host:port") flag.Parse() // Set up a connection to the server. conn, err := grpc.Dial(*address, grpc.WithInsecure()) if err != nil { log.Fatalf("did not connect: %v", err) } defer conn.Close() c := v1.NewToDoServiceClient(conn) ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() t := time.Now().In(time.UTC) reminder, _ := ptypes.TimestampProto(t) pfx := t.Format(time.RFC3339Nano) // Call Create req1 := v1.CreateRequest{ Api: apiVersion, ToDo: &v1.ToDo{ Title: "title (" + pfx + ")", Description: "description (" + pfx + ")", Reminder: reminder, }, } res1, err := c.Create(ctx, &req1) if err != nil { log.Fatalf("Create failed: %v", err) } log.Printf("Create result: <%+v>/n/n", res1) id := res1.Id // Read req2 := v1.ReadRequest{ Api: apiVersion, Id: id, } res2, err := c.Read(ctx, &req2) if err != nil { log.Fatalf("Read failed: %v", err) } log.Printf("Read result: <%+v>/n/n", res2) // Update req3 := v1.UpdateRequest{ Api: apiVersion, ToDo: &v1.ToDo{ Id: res2.ToDo.Id, Title: res2.ToDo.Title, Description: res2.ToDo.Description + " + updated", Reminder: res2.ToDo.Reminder, }, } res3, err := c.Update(ctx, &req3) if err != nil { log.Fatalf("Update failed: %v", err) } log.Printf("Update result: <%+v>/n/n", res3) // Call ReadAll req4 := v1.ReadAllRequest{ Api: apiVersion, } res4, err := c.ReadAll(ctx, &req4) if err != nil { log.Fatalf("ReadAll failed: %v", err) } log.Printf("ReadAll result: <%+v>/n/n", res4) // Delete req5 := v1.DeleteRequest{ Api: apiVersion, Id: id, } res5, err := c.Delete(ctx, &req5) if err != nil { log.Fatalf("Delete failed: %v", err) } log.Printf("Delete result: <%+v>/n/n", res5) } 复制代码
以上就是客户端所有代码,当前项目目录如下
最后一步骤是确保gPRC服务能跑起来
开启一个终端build以及run gRPC服务(将下面数据库连接的参数替代成你自己的数据库配置)
cd cmd/server go build . server.exe -grpc-port=9090 -db-host=<HOST>:3306 -db-user=<USER> -db-password=<PASSWORD> -db-schema=<SCHEMA> 复制代码
如果能看到
2018/09/09 08:02:16 starting gRPC server... 复制代码
证明我们的服务已经被启动起来了
打开另一个终端build和run gRPC客户端
cd cmd/client-grpc go build . client-grpc.exe -server=localhost:9090 复制代码
如果能看到以下信息:
2018/09/09 09:16:01 Create result: <api:"v1" id:13 > 2018/09/09 09:16:01 Read result: <api:"v1" toDo:<id:13 title:"title (2018-09-09T06:16:01.5755011Z)" description:"description (2018-09-09T06:16:01.5755011Z)" reminder:<seconds:1536473762 > > > 2018/09/09 09:16:01 Update result: <api:"v1" updated:1 > 2018/09/09 09:16:01 ReadAll result: <api:"v1" toDos:<id:9 title:"title (2018-09-09T04:45:16.3693282Z)" description:"description (2018-09-09T04:45:16.3693282Z)" reminder:<seconds:1536468316 > > toDos:<id:10 title:"title (2018-09-09T04:46:00.7490565Z)" description:"description (2018-09-09T04:46:00.7490565Z)" reminder:<seconds:1536468362 > > toDos:<id:13 title:"title (2018-09-09T06:16:01.5755011Z)" description:"description (2018-09-09T06:16:01.5755011Z) + updated" reminder:<seconds:1536473762 > > > 2018/09/09 09:16:01 Delete result: <api:"v1" deleted:1 > 复制代码
所有东西都正常运作了!
以上就是Part1的全部内容了,我们成功构建了gRPC的客户端以及服务端
Part1的源代码在 此处
接下来 Part2是讲述如何在我们本章建立的gRPC服务上增加HTTP/REST接口,敬请期待。
感谢收看!:pray::pray::pray::pray: