Service Discovery using Consul and ASP.NET Core

Go Back

Service Discovery using Consul and ASP.NET Core

by Gerade Geldenhuys Sun Jun 28, 2020

Back when I embarked on my Domain Driven Design/Microservice/Distributed systems journey I came across this neat little library to make REST calls between my services for MedPark, a sample distributed system project I am using to implement everything new I learn that is DDD related. I wrote a post on this library back when I discovered it here.

While RestEase works great at what it does, once you start doing distributed systems in every sense of the word it becomes a little trickier. When having multiple instances of an application running, having a hardcoded endpoint for a particular service in your configuration to send requests to kills the purpose of scaling, doesn’t it? For this, we can implement some kind of service discovery.

Service Discovery

Service Discovery has the ability to locate a network automatically making it so that there is no need for a long configuration set up process. Service discovery works by devices connecting through a common language on the network allowing devices or services to connect without any manual intervention. (i.e Kubernetes service discovery, AWS service discovery)

There are two types of service discovery: Server-side and Client-side. Server-side service discovery allows clients applications to find services through a router or a load balancer. Client-side service discovery allows clients applications to find services by looking through or querying a service registry, in which service instances and endpoints are all within the service registry.

Scenario

Let’s say we want to query the Catalog service and find out if a particular product is in stock. We would send this request to our API gateway and from the API we would relay that query to the Catalog service. When relaying this request, we could look up the Catalog service in some kind registry and send our request to any of the instances of the Catalog service that is healthy and available.

Consul

Consul, by HashiCorp, is a centralized service registry that enables services to discover each other by storing location information (like IP addresses) in a single registry. We will be using this service for looking up our services in a registry when communicating between services.

For local development and testing, we are using the Consul Docker image.

docker run consul -p 8500:8500

Consul Extension Methods

      
        public static IServiceCollection AddConsul(this IServiceCollection serviceCollection)
      
        {
      
            IConfiguration configuration;
      
            using (var serviceProvider = serviceCollection.BuildServiceProvider())
      
            {
      
                configuration = serviceProvider.GetService<IConfiguration>();
      
            }
      
            ConsulOptions consulConfigOptions = configuration.GetOptions<ConsulOptions>("Consul");
      
            serviceCollection.Configure<ConsulOptions>(configuration.GetSection("Consul"));
      
            serviceCollection.AddTransient<IConsulServices, ConsulServices>();
      
            serviceCollection.AddHttpClient<IConsulHttpClient, ConsulHttpClient>();
      
            return serviceCollection.AddSingleton<IConsulClient>(c => new ConsulClient(cfg =>
      
            {
      
                if (!string.IsNullOrEmpty(consulConfigOptions.Host))
      
                {
      
                    cfg.Address = new Uri(consulConfigOptions.Host);
      
                }
      
            }));
      
        }
      
        public static string UseConsul(this IApplicationBuilder app)
      
        {
      
            using (var scope = app.ApplicationServices.CreateScope())
      
            {
      
                var Iconfig = scope.ServiceProvider.GetService<IConfiguration>();
      
                var config = Iconfig.GetOptions<ConsulOptions>("Consul");
      
                var appOptions = Iconfig.GetOptions<AppOptions>("App");
      
                if (!config.Enabled)
      
                    return String.Empty;
      
                Guid serviceId = Guid.NewGuid();
      
                string consulServiceID = $"{config.Service}:{serviceId}";
      
                var client = scope.ServiceProvider.GetService<IConsulClient>();
      
                var consulServiceRistration = new AgentServiceRegistration
      
                {
      
                    Name = config.Service,
      
                    ID = consulServiceID,
      
                    Address = config.Address,
      
                    Port = config.Port
      
                };
      
                client.Agent.ServiceRegister(consulServiceRistration);
      
                return consulServiceID;
      
            }
      
        }

view raw addConsulExtension.cs hosted with ❤ by GitHub

Above are two extension methods that I am using to register my service to Consul registry. The AddConsul method adds the Consul Client and its options to the service collection, along with a custom HTTP client that will be used to make our requests between services. The UseConsul method is how we will register our application to the Consul services registry. This requires four parameters:

Name: The name of the service (i.e. Catalog-service)

ID: The ID of the service. Usually, the name of the service with a unique Id (Guid) appended to it.

Address: The address where the service will be running at.

Port: The port the service will is running on.

Sending Requests

Once we have our services registered on Consul all we need to do to request data between services are the service names. As is displayed in the screenshot above, we have 7 instances of the Basket service registered. When making our request from the API gateway to the Basket service, the API gateway does not have to know which instance to request. All we want is for Consul to send us in the direction of an instance that is healthy. Let’s look at how we can accomplish this.

Consul HTTP Client

I implemented an HTTP client to get the service we want to request from the Consul Service Registry and send our Request along. See an overview of the client implementation below:

      
        public class ConsulHttpClient : IConsulHttpClient
      
        {
      
            private readonly HttpClient _client;
      
            private IConsulClient _consulclient;
      
            public ConsulHttpClient(HttpClient client, IConsulClient consulclient)
      
            {
      
                _client = client;
      
                _consulclient = consulclient;
      
            }
      
            public async Task<T> GetAsync<T>(string serviceName, Uri requestUri)
      
            {
      
                var uri = await GetRequestUriAsync(serviceName, requestUri);
      
                var response = await _client.GetAsync(uri);
      
                if (!response.IsSuccessStatusCode)
      
                {
      
                    return default(T);
      
                }
      
                var content = await response.Content.ReadAsStringAsync();
      
                return JsonConvert.DeserializeObject<T>(content);
      
            }
      
            private async Task<Uri> GetRequestUriAsync(string serviceName, Uri uri)
      
            {
      
                //Get all services registered on Consul
      
                var allRegisteredServices = await _consulclient.Agent.Services();
      
                //Get all instance of the service went to send a request to
      
                var registeredServices = allRegisteredServices.Response?.Where(s => s.Value.Service.Equals(serviceName, StringComparison.OrdinalIgnoreCase)).Select(x => x.Value).ToList();
      
                //Get a random instance of the service
      
                var service = GetRandomInstance(registeredServices, serviceName);
      
                if (service == null)
      
                {
      
                    throw new ConsulServiceNotFoundException($"Consul service: '{serviceName}' was not found.",
      
                        serviceName);
      
                }
      
                var uriBuilder = new UriBuilder(uri)
      
                {
      
                    Host = service.Address,
      
                    Port = service.Port
      
                };
      
                return uriBuilder.Uri;
      
            }
      
            private AgentService GetRandomInstance(IList<AgentService> services, string serviceName)
      
            {
      
                Random _random = new Random();
      
                AgentService servToUse = null;
      
                servToUse = services[_random.Next(0, services.Count)];
      
                return servToUse;
      
            }
      
        }

view raw ConsulHttpClient.cs hosted with ❤ by GitHub

To better explain the code in the gist above, say we want to get a customer’s basket from the Basket service, all we need to do is pass the name of the service along to this Consul HTTP client like this basket-service. In the code above, we ask Consul for all the services that are registered with that name, then we select a random instance from the registry to send the request to. This is how we could call the basket service from our controller:

      
        [HttpGet("{customerid}")]
      
        [Cached(Constants.Day_In_Seconds)]
      
        public async Task<IActionResult> GetCustomerBasket(Guid customerid)
      
        {
      
            var basket = await _consulHttpClient.GetAsync<BasketDto>("basket-service", $"/basket/{customerid}");
      
            return Ok(basket);
      
        }

view raw basket-controller-get.cs hosted with ❤ by GitHub

The gists on this post have been chopped and screwed in a way to accommodate this article because some of it was not within the scope of giving you a high-level introduction to implementing Service discovery in your microservices based application. You can have a look at the MedPark repository for the full implementation, which also uses HealthChecks for monitoring the healthiness of a service and then deregistering the service if it is unable to handle requests.

Summary

In closing, implementing service discovery in your distributed system’s architecture reduces the technical complexity of discovering and most importantly (for me anyway) automating the process for connected services.

Thank you for reading.

	public static IServiceCollection AddConsul(this IServiceCollection serviceCollection)
	{
	IConfiguration configuration;
	using (var serviceProvider = serviceCollection.BuildServiceProvider())
	{
	configuration = serviceProvider.GetService<IConfiguration>();
	}
	ConsulOptions consulConfigOptions = configuration.GetOptions<ConsulOptions>("Consul");

	serviceCollection.Configure<ConsulOptions>(configuration.GetSection("Consul"));
	serviceCollection.AddTransient<IConsulServices, ConsulServices>();
	serviceCollection.AddHttpClient<IConsulHttpClient, ConsulHttpClient>();

	return serviceCollection.AddSingleton<IConsulClient>(c => new ConsulClient(cfg =>
	{
	if (!string.IsNullOrEmpty(consulConfigOptions.Host))
	{
	cfg.Address = new Uri(consulConfigOptions.Host);
	}
	}));
	}

	public static string UseConsul(this IApplicationBuilder app)
	{
	using (var scope = app.ApplicationServices.CreateScope())
	{
	var Iconfig = scope.ServiceProvider.GetService<IConfiguration>();

	var config = Iconfig.GetOptions<ConsulOptions>("Consul");
	var appOptions = Iconfig.GetOptions<AppOptions>("App");

	if (!config.Enabled)
	return String.Empty;

	Guid serviceId = Guid.NewGuid();
	string consulServiceID = $"{config.Service}:{serviceId}";

	var client = scope.ServiceProvider.GetService<IConsulClient>();

	var consulServiceRistration = new AgentServiceRegistration
	{
	Name = config.Service,
	ID = consulServiceID,
	Address = config.Address,
	Port = config.Port
	};

	client.Agent.ServiceRegister(consulServiceRistration);

	return consulServiceID;
	}
	}

	public class ConsulHttpClient : IConsulHttpClient
	{
	private readonly HttpClient _client;
	private IConsulClient _consulclient;

	public ConsulHttpClient(HttpClient client, IConsulClient consulclient)
	{
	_client = client;
	_consulclient = consulclient;
	}

	public async Task<T> GetAsync<T>(string serviceName, Uri requestUri)
	{
	var uri = await GetRequestUriAsync(serviceName, requestUri);

	var response = await _client.GetAsync(uri);

	if (!response.IsSuccessStatusCode)
	{
	return default(T);
	}

	var content = await response.Content.ReadAsStringAsync();

	return JsonConvert.DeserializeObject<T>(content);
	}

	private async Task<Uri> GetRequestUriAsync(string serviceName, Uri uri)
	{
	//Get all services registered on Consul
	var allRegisteredServices = await _consulclient.Agent.Services();

	//Get all instance of the service went to send a request to
	var registeredServices = allRegisteredServices.Response?.Where(s => s.Value.Service.Equals(serviceName, StringComparison.OrdinalIgnoreCase)).Select(x => x.Value).ToList();

	//Get a random instance of the service
	var service = GetRandomInstance(registeredServices, serviceName);

	if (service == null)
	{
	throw new ConsulServiceNotFoundException($"Consul service: '{serviceName}' was not found.",
	serviceName);
	}

	var uriBuilder = new UriBuilder(uri)
	{
	Host = service.Address,
	Port = service.Port
	};

	return uriBuilder.Uri;
	}

	private AgentService GetRandomInstance(IList<AgentService> services, string serviceName)
	{
	Random _random = new Random();

	AgentService servToUse = null;

	servToUse = services[_random.Next(0, services.Count)];

	return servToUse;
	}
	}

	[HttpGet("{customerid}")]
	[Cached(Constants.Day_In_Seconds)]
	public async Task<IActionResult> GetCustomerBasket(Guid customerid)
	{
	var basket = await _consulHttpClient.GetAsync<BasketDto>("basket-service", $"/basket/{customerid}");

	return Ok(basket);
	}

About Me: