Request signatures with tonic + tower

I’m working on an infrastructure application that provides a GRPC API to its clients. This will be a high throughput system, so we are building it in Rust using tonic and tokio to build the gRPC API. I wanted to implement request authentication for this system so that the service was resilient to Server-Side-Request-Forgery, and unauthenticated requests.

Request signature authentication

I wanted to implement HMAC signature based authentication. I’ve used this in the past to secure internal APIs, and it is a simple but effective solution for scenarios where you control both ends of the network request. The general idea is that both your client and server applications are provided a shared secret. This secret is used to sign and verify requests from authorized clients. The client is responsible for generating and including a request signature header in every request. While the server is responsible for validating that the signature matches the request payload. For this application, we would combine the request URL and request body to create a signature. In python my client code looks like:

Show Plain Text
  1. # This logic lives inside a gRPC interceptor
  2. # request is a protocol buffer message
  3. request_body = request.SerializeToString()
  4. method = client_call_details.method.encode("utf-8")
  5.  
  6. signing_payload = method + b":" + request_body
  7. signature = hmac.new(self._secret, signing_payload, hashlib.sha256).hexdigest()

Then on the server side, if we can read the request body into Bytes, we can compare the HMAC value with the request data:

Show Plain Text
  1. use anyhow::Context;
  2. use bytes::Bytes;
  3. use hmac::{Hmac, Mac};
  4. use sha2::Sha256;
  5. use std::mem;
  6.  
  7. // 5 bytes
  8. const HEADER_SIZE: usize =
  9.     // compression flag
  10.     std::mem::size_of::<u8>() +
  11.     // data length
  12.     std::mem::size_of::<u32>();
  13.  
  14. fn validate_signature(
  15.     secret: &[String],
  16.     req_head: &http::request::Parts,
  17.     req_body: Bytes,
  18. ) -> anyhow::Result<Bytes> {
  19.     if secret.is_empty() {
  20.         return Ok(req_body);
  21.     }
  22.  
  23.     let signature = req_head
  24.         .headers
  25.         .get("signature")
  26.         .context("missing signature header")?
  27.         .to_str()
  28.         .unwrap_or_default();
  29.  
  30.     // Signature in the request headers is hex encoded
  31.     let signature = hex::decode(signature).unwrap_or_default();
  32.  
  33.     // gRPC messages are prefix-length encoded with a 5 byte header
  34.     // https://github.com/hyperium/tonic/blob/master/tonic/src/codec/mod.rs#L93
  35.     // Slice off the header as it won't be included in the signature.
  36.     let trimmed_body = &req_body[HEADER_SIZE..];
  37.  
  38.     for possible_key in secret {
  39.         let mut hmac = Hmac256::new_from_slice(possible_key.as_bytes()).unwrap();
  40.         hmac.update(req_head.uri.path().as_bytes());
  41.         hmac.update(b":");
  42.         hmac.update(trimmed_body);
  43.  
  44.         if hmac.verify_slice(&signature[..]).is_ok() {
  45.             return Ok(req_body);
  46.         }
  47.     }
  48.  
  49.     anyhow::bail!("Invalid request signature")
  50. }

Using request signatures is an resource efficient, low complexity solution to request authorization. It does have some weaknesses though that are worth being aware of. Requests that have low variance in their parameters will generate predictable patterns in signatures. You can combat this by including additional inputs to the HMAC like a timestamp, or request identifier that uses an autoincrement, or UUID. More advanced solutions like service mesh authentication have fewer weaknesses but have higher complexity, as you need to operate additional load balancers and a certificate authority.

Tower middleware

I’ve built the client and server logic for request signatures in Python before and it was an afternoon of work once you factor in tests and documentation. Since my server application is built with tonic, tower and tokio, I wanted to use those to implement request signatures. A tower middleware would ensure that all requests had authentication applied. The tower documentation has some good examples to get started with:

Show Plain Text
  1. use anyhow::Context;
  2. use std::task::{self, Poll};
  3. use tower::{Layer, Service};
  4.  
  5. /// Tower layer to connect authentication logic with the grpc server
  6. #[derive(Debug, Clone, Default)]
  7. pub struct AuthLayer {
  8.     /// A list of secrets enables key rotation.
  9.     pub shared_secret: Vec<String>,
  10. }
  11.  
  12. impl AuthLayer {
  13.     pub fn new(secret: Vec<String>) -> Self {
  14.         Self {
  15.             shared_secret: secret,
  16.         }
  17.     }
  18. }
  19.  
  20. impl<Inner> Layer<Inner> for AuthLayer {
  21.     type Service = AuthService<Inner>;
  22.  
  23.     fn layer(&self, service: Inner) -> Self::Service {
  24.         AuthService::new(self.shared_secret.clone(), service)
  25.     }
  26. }
  27.  
  28. /// Tower service that provides a home for the authentication logic
  29. #[derive(Debug, Clone)]
  30. pub struct AuthService<Inner> {
  31.     secret: Vec<String>,
  32.     inner: Inner,
  33. }
  34.  
  35. impl<Inner> AuthService<Inner> {
  36.     pub fn new(secret: Vec<String>, inner: Inner) -> Self {
  37.         Self { secret, inner }
  38.     }
  39. }
  40.  
  41. impl<Inner, ReqBody> Service<http::Request<ReqBody>> for AuthService<Inner>
  42. where
  43.     Inner: Service<http::Request<ReqBody>>
  44. {
  45.     type Response = Inner::Response;
  46.     type Error = Inner::Error;
  47.     type Future = Inner::Future;
  48.  
  49.     fn poll_ready(&mut self, cx: &mut task::Context<'_>) -> Poll<Result<(), Self::Error>> {
  50.        self.inner.poll_ready(cx)
  51.    }
  52.  
  53.    fn call(&mut self, req: http::Request<ReqBody>) -> Self::Future {
  54.        self.inner.call(req)
  55.    }
  56. }

From what I was able to gather from the tower docs, I would need both an Layer and Service. The Layer seems to act as a factory for the Service. The docs have a few good examples of building middleware for timeouts and backpressure, but no examples where the request body was read. Reading the request headers was easy enough, but reading the body was an entirely different story. I had hoped I could write something like this logic:

Show Plain Text
  1. let (parts, body) = req.into_parts();
  2. let body_bytes = body.collect().await.unwrap().to_bytes();

While this code would compile fine in a test where I could control the types carefully. Doing this in the context of my middleware was not straightforward as the compiler was not at all pleased.

Reading the request data is an async operation, as we may need to wait for more data to arrive on the socket. While Service::call() requires a Future to be returned, I had the hardest time trying to figure out how to connect the body reading future with the rest of my authentication logic. I would also need a way to short-circuit the request when authentication failed. I started off trying to sketch out how to integrate validate_signature into my Service::call() method:

Show Plain Text
  1. fn call(&mut self, req: http::Request<ReqBody>) -> Self::Future {
  2.     let secret = self.secret.clone();
  3.     // Can't figure out how to read the body yet.
  4.     let (parts, _body) = req.into_parts();
  5.     let body_bytes = Bytes::from("request data");
  6.  
  7.     match validate_signature(&secret, &parts, body_bytes) {
  8.         Ok(body) => {
  9.             // reconstruct a request with the bytes we read from the request.
  10.             let new_req = http::Request::from_parts(parts, body);
  11.  
  12.             self.inner.call(new_req)
  13.         }
  14.         Err(error) => {
  15.             tracing::error!("GRPC Authentication error: {error}");
  16.             let response =
  17.                 tonic::Status::unauthenticated("Authentication failed").into_http();
  18.             Ok(response)
  19.         }
  20.     }
  21. }

This code didn’t work yet, though as the compiler was upset about types. There were a few problems:

1. My success case was returning the wrapped Service result, which was a wide generic. Because the request’s body was still a generic – ReqBody – it could be any type.
2. The return of validate_signature was Result<Bytes, Error>. The compiler inferred that the request must be at least UnsyncBoxBody which is an opaque type, that I couldn’t figure out how to interact with.
3. The error case isn’t returing a future, and I hadn’t figured out how to make it do that.

I’m still learning rust, and didn’t fully understand how to go about solving these problems. After spending a few hours trying to figure it out without success, I turned to the tokio discord. Thankfully AlphaKeks helped me better understand what I was missing.

Traits and generics need boundaries

In Rust there is no inheritance. There are also no interfaces as you would have them in PHP, Python, or Java. Instead rust uses ‘traits’ which work similar to golang’s interfaces, or to TypeScript’s structural typing. Traits describe a set of methods that can be implemented by different structs. Functions can have parameter types that require one or more traits to be implemented. Because traits often involve generics, those generics need to be be narrowed or defined for implementations. Each trait + generic parameter type can have a different method implementation, which is an interesting feature of the rust trait system.

Because my AuthService trait implementation interacts with the ReqBody generic, I needed to define the trait ‘boundaries’ for the interactions I wanted to do with the request. The whole concept of trait boundaries took me a while to understand. What helped make it click was incrementally rebuilding the suggested solution piece by piece and fixing compiler errors along the way. When I took it slow and carefully read the compiler errors, I was able to incrementally narrow the bounds of the overly general generics and reduce the compiler errors.

The Future

The final boss was the future type. I took the advice and used Box::pin with an async block to make a BoxFuture. I had not encountered ‘pinning’ before, and it took some time for me to understand. The gist of it is for futures to work in rust, they can’t have their memory addresses ‘moved’ as that would invalidate the self-references needed for future state-machines. The BoxFuture type provides a generic ‘box’ that any type can be stored in. I figured out that I could narrow the request body to be UnsyncBoxBody<Bytes, tonic::Status> which solved the last of my compiler errors. The end result is:

Show Plain Text
  1. type TonicBody = UnsyncBoxBody<Bytes, tonic::Status>;
  2.  
  3. impl<Inner> Service<http::Request<TonicBody>> for AuthService<Inner>
  4. where
  5.     // Narrow the generics on `Service` to match the interfaces I needed to use.
  6.     Inner: Service<http::Request<TonicBody>, Response = http::Response<TonicBody>>
  7.         + Clone
  8.         + Send
  9.         + 'static,
  10.    Inner::Future: Send,
  11. {
  12.    type Response = Inner::Response;
  13.    type Error = Inner::Error;
  14.    type Future = BoxFuture<'static, Result<Self::Response, Self::Error>>;
  15.  
  16.     fn poll_ready(&mut self, cx: &mut task::Context<'_>) -> Poll<Result<(), Self::Error>> {
  17.        self.inner.poll_ready(cx)
  18.    }
  19.  
  20.    fn call(&mut self, req: http::Request<TonicBody>) -> Self::Future {
  21.        let secret = self.secret.clone();
  22.        // Clone & swapping is necessary to access `inner` within the async block.
  23.        let mut inner = self.inner.clone();
  24.        mem::swap(&mut inner, &mut self.inner);
  25.  
  26.        // Because we need to read the request body we need an async block
  27.        // and to pin the futures.
  28.        Box::pin(async move {
  29.            let (parts, body) = req.into_parts();
  30.            let body_bytes = body.collect().await.unwrap().to_bytes();
  31.  
  32.            match validate_signature(&secret, &parts, body_bytes) {
  33.                Ok(body) => {
  34.                    // reconstruct a request with the bytes we read from the request.
  35.                    // tonic::body::boxed satifies the TonicBody trait bounds.
  36.                    let new_body = Full::new(body);
  37.                    let boxed_body = tonic::body::boxed(new_body);
  38.                    let new_req = http::Request::from_parts(parts, boxed_body);
  39.  
  40.                    inner.call(new_req).await
  41.                }
  42.                Err(error) => {
  43.                    tracing::error!("GRPC Authentication error: {error}");
  44.                    let response =
  45.                        tonic::Status::unauthenticated("Authentication failed").into_http();
  46.                    Ok(response)
  47.                }
  48.            }
  49.        })
  50.    }
  51. }

What I learned

I have a few takeaways from this experience:

1. Rust traits are very powerful and enable very flexible interfaces to be designed. They can be difficult to consume until you have done it a few times.
2. Associated trait types are complicated and I should have taken a step back and read the documentation more thoroughly earlier on.
3. Incrementally building code with complex types should be done as a series of very small steps where feedback from cargo build is incorporated and compiler errors resolved incrementally. Doing big changes and then trying to resolve all the compiler errors was very challenging for me as a novice rust developer.

Comments

There are no comments, be the first!

Have your say: